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Life’s most important questions are, for the most part, 
nothing but probability problems. 


Pierre Simon de Laplace 


Preface 


This book is based on the lecture notes that I have been using since 1988 
for the course entitled Processus stochastiques at the Ecole Polytechnique de 
Montréal. This course is mostly taken by students in electrical engineering 
and applied mathematics, notably in operations research, who are generally 
at the master’s degree level. Therefore, we take for granted that the reader 
is familiar with elementary probability theory. However, in order to write a 
self-contained book, the first chapter of the text presents the basic results in 
probability. 

This book aims at providing the readers with a reference that covers the 
most important subjects in the field of stochastic processes and that is ac- 
cessible to students who don’t necessarily have a sound theoretical knowledge 
of mathematics. Indeed, we don’t insist very much in this volume on rigor- 
ous proofs of the theoretical results; rather, we spend much more time on 
applications of these results. 

After the review of elementary probability theory in Chapter 1, the remain- 
der of this chapter is devoted to random variables and vectors. In particular, 
we cover the notion of conditional expectation, which is very useful in the 
sequel. 

The main characteristics of stochastic processes are given in Chapter 2. 
Important properties, such as the concept of independent and stationary in- 
crements, are defined in Section 2.1. Next, Sections 2.2 and 2.3 deal with 
ergodicity and stationarity, respectively. The chapter ends with a section on 
Gaussian and Markovian processes. 

Chapter 3 is the longest in this book. It covers the cases of both discrete- 
time and continuous-time Markov chains. We treat the problem of calculating 
the limiting probabilities of the chains in detail. Branching processes and birth 
and death processes are two of the particular cases considered. The chapter 
contains nearly 100 exercises at its end. 

The Wiener process is the main subject of Chapter 4. Various processes 
based on the Wiener process are presented as well. In particular, there are sub- 
sections on models such as the geometric Brownian motion, which is very im- 
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portant in financial mathematics, and the Ornstein—Uhlenbeck process. White 
noise is defined, and first-passage problems are discussed in the last section 
of the chapter. 

In Chapter 5, the Poisson process, which is probably the most important 
stochastic process for students in telecommunications, is studied in detail. 
Several generalizations of this process, including nonhomogeneous Poisson 
processes and renewal processes, can be found in this chapter. 

Finally, Chapter 6 is concerned with the theory of queues. The models 
with a single server and those with at least two servers are treated separately. 
In general, we limit ourselves to the case of exponential models, in which both 
the times between successive customers and the service times are exponential 
random variables. This chapter then becomes an application of Chapter 3 
(and 5). | 

In addition to the examples presented in the theory, the book contains ap- 
proximately 350 exercises, many of which are multiple-part problems. These 
exercises are all problems given in exams or homework and were mostly cre- 
ated for these exams or homework. The answers to the even-numbered prob- 
lems are given in Appendix B. 

Finally, it is my pleasure to thank Vaishali Damle, Julie Park, and Eliza- 
beth Loew from Springer for their work on this book. 


Mario Lefebvre 
Montréal, November 2005 
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Review of Probability Theory 


1.1 Elementary probability 


Definition 1.1.1. A random experiment is an experiment that can be re- 
peated under the same conditions and whose result cannot be predicted with 
certainty. 


Example 1.1.1. We consider the following three classical experiments: 

E): a coin is tossed three times and the number of “tails” obtained is recorded; 
Eo: a die is thrown until a “6” appears, and the number of throws made is 
counted; 

£3: a number is taken at random in the interval (0, 1). 


Remark. A closed interval will be denoted by [a,b], whereas we write (a,b) in 
the case of an open interval, rather than Ja, bf, as some authors write. 


Definition 1.1.2. The sample space S of a random experiment is the set 
of all possible outcomes of this experiment. 


Example 1.1.2. The sample spaces that correspond to the random experiments 
in the example above are the following: 


Sy = 1 L23 
So = TOT 
S$, ={0;1). 


Definition 1.1.3. An event is a subset of the sample space S. In particu- 
lar, each possible outcome of a random experiment is called an elementary 
event. 


The number of elementary events in a sample space may be finite (51), 
countably infinite (S2), or uncountably infinite (S3). 
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S 


C 


Fig. 1.1. Venn diagram. 


We often use Venn! diagrams in elementary probability: the sample space 
S is represented by a rectangle and the events A, B, C, etc. by circles that 
overlap inside the rectangle (see Fig. 1.1). 


Example 1.1.3. We can define, in particular, the following events with respect 
to the sample spaces associated with the random experiments of Example 
ElL 

A; = “tails” is obtained only once, that is, Ay = {1}; 

Ag = six or seven throws are made to obtain the first “6,” that is, Ag = {6,7}; 
A3 = the number taken at random is smaller than 1/2, that is, A3 = [0, 1/2). 


Notations 
Union: AU B (corresponds to the case when we seek the probability that one 
event or another one occurred, or that both events occurred). 


Intersection: AM B or AB (when we seek the probability that an event and 
another one occurred). If two events are incompatible (or mutually exclusive), 
then we write that AN B = @ (the empty set). 

Complement: A‘ (the set of elementary events that do not belong to A). 


Inclusion: A C B (when all the elementary events that belong to A also 
belong to B). | 


Definition 1.1.4. A probability measure is a function P of the subsets of 
a sample space S, associated with a random experiment E, that possesses the 
following properties: 


Axiom I: P[A] >0 VACS; 


Axiom II: P[S] = 1; 


* John Venn, 1834-1923, was born and died in England. He was a mathemati- 
cian and priest. He taught at the University of Cambridge and worked in both 
mathematical logic and probability theory. 
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Axiom ITI: Jf Aj, Ao,... is an infinite sequence of events that are all incom- 
patible when taken two at a time, then 


P| U an| = yo Pia (1.1) 


In particular, 


P[AU B] = P[A] + P|B] if ANB=9 (1.2) 


If the number n of elementary events is finite and if these events are 
equiprobable (or equally likely), then we may write that 


n(A) 


PA n 


(1.3) 
where n(A) is the number of elementary events in A. However, in general, 
the elementary events are not equiprobable. For instance, the four elementary 
events of the sample space S; in Example 1.1.2 have the following probabil- 
ities: P[{O}] = PH3} = 1/8 and PH1} = P[{2}] = 3/8 (if we assume that 
the coin is fair), and not P|{k}] = 1/4, for k = 0,1,2,3. 


Remark. It is said that the French mathematician d’Alembert? believed that 
when a fair coin is tossed twice, then the probability of getting one “tail” and 
one “head” is equal to 1/3. His reasoning was based on the fact that there 
are three possible outcomes in this random experiment: getting two “tails”; 
two “heads”; or one “tail” and one “head.” It is easy to determine that the 
probability of obtaining one “tail” and one “head” is actually 1/2, because 
here there are four equiprobable elementary events: Hı Hə (that is, “heads” 
on the first and on the second toss); H,T>; T H2; and T,T»>. Finally, the event 
A: getting one “tail” and one “head” corresponds to two elementary events: 
HiT and T Hə. 


Proposition 1.1.1. We have 
1) P(A} = 1 — PIA] VACS. 
2) For all events A and B, 
P [AU B] = P[A] + P[B] — P[AB| (1.4) 


3) For any three events A, B, and C, 


P[AU BUC] = P[A] + PIB] + P[C] — P[AB] — PIAC) - P[BC] + srl 
1.5 


* Jean Le Rond d’Alembert, 1717-1783, was born and died in France. He was a 
prolific mathematician and writer. He published books, in particular, on dynamics 
and on the equilibrium and motion of fluids. 
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Proposition 1.1.2. [fk objects are taken at random among n distinct objects 
and if the order in which the objects were drawn does matter, then the number 
of different permutations that can be obtained is given by 


nmxnx...xn=n* (1.6) 


if the objects are taken with replacement, and by 


n! A 


n x(n—=1)x... x Ea ay == P? fork =0,1,...,n 
(1.7) 
when the objects are taken without replacement. 
Remark. If, among the n objects, there are n; of type i, where 1 = 1,2,...,9, 


then the number of different permutations of the entire set of objects is given 
by (the multinomial coefficients) 
se (1.8) 
ning! +--+ nj! 
Example 1.1.4. The combination of a certain padlock is made up of three 
digits. Therefore, theoretically there are 10? = 1000 possible combinations. 
However, in practice, if we impose the following constraint: the combination 
cannot be made up of two identical consecutive digits, then the number of 
possible combinations is given by 10 x 9 x 9 = 810. This result can also be 
obtained by subtracting from 1000 the number of combinations with at least 
two identical consecutive digits, namely, 10 (with three identical digits) + 
10x1x9+10x9-x 1 (with ezactly two identical digits, either the first two 
or the last two digits). 


Proposition 1.1.3. If k objects are taken, at random and without replace- 
ment, among n distinct objects and if the order in which the objects were 
drawn does not matter, then the number of different combinations that can 
be obtained is given, fork =0,1,... ,n, by 


nx (n-1)x...x[n-(k-1) n! n 

k! k!(n — k)! k s oe 
Remark. lf the objects are taken with replacement, then the number of different 
combinations is Ca In this case, k may take any value in N° := {0,1,...}. 


Remark. In the preceding example, according to the common use, the word 
“combinations” was used for a padlock. However, they were indeed “permu- 
tations.” 
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Example 1.1.5. In a given lottery, 6 balls are drawn at random and without 
replacement among 49 balls numbered from 1 to 49. We win a prize if the 
combination that we chose has at least three correct numbers. Let 

F = we win a prize 


and 
Fy, = our combination has exactly k correct numbers, for k = 0,1,... ,6. 
We have 
2 2 (°) ( 43 ) 
PIF) =1-)°P[R)=1-)°“* 
_4 (1 - 6,096, 454) + (6 - 962, 598) + (15 - 123, 410) 
7 13, 983, 816 
13, 723, 192 
= | — —__——__ ~ | — 0.9814 = 0.0186 
13, 983, 816 : 


Notation. The expression P[A | B] denotes the probability of the event A, 
given that (or knowing that, or simply if) the event B has occurred. 


Definition 1.1.5. We set 


PJAN B] 


PIA| B| = -Sr 


if PIB] > 0 (1.10) 


Proposition 1.1.4. (Multiplication rule) We have 


P[AN B] = PA | B] x P|B} = PIB | A] x PIA] if P[A]P|B]>0 (1.11) 


Example 1.1.6. In the preceding example, let 
Fk = the number of the kth ball that is drawn is part of our combination. 
Generalizing the multiplication rule, we may write that 


PIP, OF) F3| = PIF; | Fi Nn Fo] PLF | Pil P[Fi] 
O4 5. 6 120 


we eet ag ee IT 
47” 48” 49 110,544 


Definition 1.1.6. The events Bi, B2,... , Bn constitute a partition of the 
sample space S if 


i) BB; =O0Vis¥s, 
st) |) ge BESS, 
i) PIB > 0- Jor k= ND ie om: 
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If By, Bo,... , Bn is a partition of S, then we may write, for any event A, 
that 


A=(ANB,)U(AN Bo) U...U(AN Bn) (1.12) 


where (AN Bi) N (AN B;) = V i # j. Making use of Axiom III in the 
definition of the function P, p. 2, we obtain the following result. 


Proposition 1.1.5. (Total probability rule) If A C S and the events By, 


Bo, ..., Bn form a partition of S, then 
P[A] = X` PIAN By] = X PIA | Bel P[Be] (1.13) 
k=1 k=1 


Finally, we deduce from the total probability rule and from the formula 


P[B | AJP[A] 


PIA| B| = — pp 


if P[A]P[B] > 0 (1.14) 


the result known as Bayes’? rule (or formula, or also theorem). 
Proposition 1.1.6. (Bayes’ rule) Jf P|A] > 0, then 


PIA | Bi] PÍB;] 


P(B; | A] = = PIA | BP IBal 


| ee ae 2 sn (1.15) 


where Bı, B2,... , Bn is a partition of S. 


Example 1.1.7. In a certain institution, 80% of the teaching staff are men. 
Moreover, 80% of the male teachers hold a Ph.D. and 90% of the female 
teachers hold a Ph.D. A teacher from this institution is taken at random. Let 
F = this teacher is a woman 
and 
D = this teacher holds a Ph.D. 
We may write, by the total probability rule, that 


PD] = P[D | F|P[F] + P[D | F°|P|F°] = 0.9 x 0.2 + 0.8 x 0.8 = 0.82 
Furthermore, we have 


pir | p= PP] EU 00 0 eee 


P[D] -0.82 
> The Reverend Thomas Bayes, 1702-1761, was born and died in England. His 


works on probability theory were published in a posthumous scientific paper in 
1764, 


1.1 Elementary probability T 
Definition 1.1.7. Two events, A and B, are said to be independent if 


PIAN B] = P[A|P[B] (1.16) 


Remark. Let C be an event such that P[C] > 0. We say that A and B are 
conditionally independent with respect to C if 


PJAN B | C] = P[A | C]PIB | C] (1.17) 


If, in particular, A and C, or B and C, are incompatible, then A and B are 
conditionally independent with respect to C, whether they are independent 
or not. Moreover, A and B can be independent, but not conditionally inde- 
pendent with respect to C. For instance, this may be the case if A and C, and 
B and C, are not incompatible, but AN BNC =f. 


For events A and B such that P[A] x P[B] > 0, the next proposition could 
serve as a more intuitive definition of independence. 


Proposition 1.1.7. Two events, A and B, having a positive probability are 
independent if and only if 


P|A|B]=P[A] or P[B|A]= P|B] (1.18) 


Proposition 1.1.8. If A and B are independent, then so are AC and B, A 
and B°, and AF and B®. 


Remark. The preceding proposition is obviously false if we replace the word 
“independent” by “incompatible” (and if A and B are not the sample space 


S). 


All that remains to do is to generalize the notion of independence to any 
number of events. 


Definition 1.1.8. The events A;,Ao,...,An are said to be independent 1f 
we may write that 


k 
Plia NAg Aal] PTA] (1.19) 


j=l 


for k = 2,3,...,n, where the events A;, E€ {A1,... , An} V j are all different. 


Remark. If the preceding definition is satisfied (at least) for k = 2, we say that 
the events A;, Á2,... , Án are pairwise independent. 
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tt 


(a) (b) 


Fig. 1.2. Examples of (a) a series system and (b) a parallel system. 


Example 1.1.8. A given system is made up of n components placed in series 
and that operate independently from one another [see Fig. 1.2 (a)|. Let 
F = the system is functioning at time to 
and 
Fk = component k is functioning at time to, for k = 1,...,n. 
We have 


PIF] = PIF N Fa... Fy] =E |] PIF] 
k=1 


Remark. To help out the reader, the justification of the equality, as here by 
independence (abbreviated as ind.), is sometimes placed above the equality 
sign. 

When the components are placed in parallel, we may write that 


P|F] =1— P|F}] =1-P[Fon...n Fy! i Tja — P[F;}) 
k=1 


1.2 Random variables 


Definition 1.2.1. A random variable (r.v.) is a function X that associates 
a real number X(s) = x with each element s of S, where S is a sample space 
associated to a random experiment E. We denote by Sx the set of all possible 
values of X (see Fig. 1.3). 


Remark. The reason for which we introduce the concept of a random variable 
is that the elements s of the sample space S can be anything, for example, 
a color or a brand of object. Since we prefer to work with real numbers, we 
transform (if needed) each s into a real number x = X(s). 
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S A(s) = x 


Fig. 1.3. Graphical representation of a random variable. 


Example 1.2.1. Consider the random experiment E3 in Example 1.1.1 that 
consists of taking a number at random in the interval (0,1). In this case, the 
elements of S are already real numbers, so that we can define the r.v. X that 
is simply the number obtained. That is, X is the identity function which with 
s associates the real number s. 

We can define other random variables on the same sample space S, for 
example: 


yi! if the number obtained is smaller than 1/2 
~ | 0 otherwise 


(called the indicator variable of the event A: the number obtained is smaller 
than 1/2) and Z(s) = Ins; that is, Z is the natural logarithm of the number 
taken at random in the interval (0, 1). 

We have 


Sx =S= (0,1), Sy = {0,1}, and Sz = (—oo, 0) 
Definition 1.2.2. The distribution function of the r.v. X is defined by 
Fy(z)=P[X <z) VaeR (1.20) 
Properties. i) 0 < Fy(x) < 1. 
MH) ities dx = and Miia rE L 
iii) If zı < x2, then Fx (x1) < Fx (x2). 


iv) The function Fx is (at least) right-continuous: 


Fro = Fx(z*):= lim Fy (z+) (1.21) 


Proposition 1.2.1. We have 
i) Pla < X <b) = Fx(b) — Fx(a), 
t) PIX = 2) = Fx(x) — Fy(x~), where F(z) := lim: jo. Fx (x — €). 


Remark. Part ii) of the preceding proposition implies that P|X = zx] = 0 for 
all x where the function Fy (zx) is continuous. 
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Definition 1.2.3. The conditional distribution function of the r.v. X, 
given an event A for which PIA] > 0, is defined by 


PHX <a}NAl 


Fx(x | A) = PÍA] 


(1.22) 


Remark. A marginal distribution function Fy (x) is simply the particular case 
of the preceding definition where the event A is the sample space S. 


Example 1.2.2. The distribution function of the r.v. X in Example 1.2.1 is 
given by 

0Oifx <0 

ix(QaPi xs gi a< ero ead 

Lied 
It is easy to check that this function possesses all the properties of a distribu- 
tion function. In fact, it is continuous for all real z, so that we can state that 
the number that will be taken at random in the interval (0,1) had, a priori, 
a zero probability of being chosen, which might seem contradictory. However, 
there are so many real numbers in the interval (0,1) that if we assigned a 
positive probability to each of them, then the sum of all these probabilities 
would diverge. 

Next, consider the event A: the number obtained is smaller than 1/2. Since 

P[A] = 1/2 > 0, we calculate 


Fy(x| A)=P[X <x|X <1/2]}=2P[X <a] if0< z< 1/2 
so that we may write that 


0 ifx <0 
Fy(x| A) = < 22 if0< a2 <1/2 
Lee 172 
Definition 1.2.4. If the set Sx of values that the r.v. X can take is finite or 
countably infinite, we say that X is a discrete r.v. or an r.v. of discrete 


type. 


Definition 1.2.5. The probability mass function of the discrete r.v. X is 
defined by 


px(zk) = PIX = Tkl V Tk E Sx (1.23) 


Remarks. i) Properties: a) px (zk) > 0 Y zk; b) Yon esy PX (Tk) = 1. 
ii) We may write that 


Fx(t)= $ px(zk)u(z-— zr) (1.24) 
TkESX 
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where u(x) is the Heaviside* function, defined by 


Oifx<0 


iii) Generalization: the conditional probability mass function of X, given an 
event A having a positive probability, is defined by 


PHX =x} A] 


PIA (1.26) 


px(z| A) = 
Example 1.2.3. An urn contains five white balls and three red balls. We draw 
one ball at a time, at random and without replacement, until we obtain a white 


ball. Let X be the number of draws needed to end the random experiment. 
We find that 





and 





Finally, let A: the first white ball is obtained after at most two draws. We 


have P[A] = Fx(2)= 2+ x 2 = 22 and 





Important discrete random variables 


i) Bernoulli? distribution: we say that X has a Bernoulli distribution with 
parameter p, where p is called the probability of a success, if 


px(x)=p*(1—p)'* forz =0 and 1 (1.27) 


“ Oliver Heaviside, 1850-1925, who was born and died in England, was a physicist 
who worked in the field of electromagnetism. He invented operational calculus to 
solve ordinary differential equations. 

5 Jacob (or Jacques) Bernoulli, 1654-1705, was born and died in Switzerland. His 
important book on probability theory was published eight years after his death. 
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Remark. The term (probability) distribution is used to designate the set of 
possible values of a discrete random variable, along with their respective prob- 
abilities given by the probability mass function. By extension, the same term 
will be employed in the continuous case (see p. 13). 


ii) Binomial distribution with parameters n and p: Sx = {0,1,...,n} and 


px (x) = « p*(1—p)""* (1.28) 


We write X ~ B(n,p). Some values of its distribution function are given in 
Table 6.4, p. 358. 


iii) Geometric distribution with parameter p: Sx = {1,2,...} and 


px(x).=(1—p)""'p (1.29) 
We write X ~ Geom(p). 
iv) Poisson® distribution with parameter \ > 0: Sx = {0,1,...} and 


AZ 
px(z) = eò (1.30) 


We write X ~ Poi(A). Its distribution function is given, for some values of A, 
in Table 6.4, p. 361. 


Poisson approximation. If n is large enough (>20) and p is sufficiently 
small (<0.05), then we may write that 


PIB pi =k =P Poal =p) =k) for k= 0an (1.31) 
If the parameter p is greater than 1/2, we proceed as follows: 


P[B(n,p) = k] = P[B(n, 1 — p) = n — k] 
~ P[Poi(\ = n(1 — p)) = n — k] (1.32) 


When p > 1/2, we also have 


P[B(n, p) < k] = P[B(n,1 — p) > n — k] 
œ P[Poi(A = n(1 — p)) > n— k] (1.33) 


8 Siméon Denis Poisson, 1781-1840, was born and died in France. He first stud- 
ied medicine and, from 1798, mathematics at the École Polytechnique de Paris, 
where he taught from 1802 to 1808. His professors at the Ecole Polytechnique 
were, among others, Laplace and Lagrange. In mathematics, his main results 
were his papers on definite integrals and Fourier series. The Poisson distribution 
appeared in his important book on probability theory published in 1837. He also 
published works on mechanics, electricity, magnetism, and astronomy. His name 
is associated with numerous results in both mathematics and physics. 
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Example 1.2.4. Suppose that we repeat the random experiment in Example 
1.2.3 20 times and that we count the number of times, which we denote by 
X, that the first white ball was obtained on the fourth draw. We may write 
that X ~ B(n = 20, p = 1/56). We calculate 


ceva 1 5549 
PIX <1)= (5) +20 (=) (5) ~ 0.9510 


The approximation with a Poisson distribution gives 
P(X <1) ~ PIY <1], where Y ~ Poi(20/56) 
5 
= ¢~5/14 ae S/E ~ 9.9496 
e TE vi 


Definition 1.2.6. A continuous random variable X is an r.v. that can 
take an uncountably infinite number of values and whose distribution function 
Fy is continuous. 


Definition 1.2.7. The (probability) density function of the continuous r.v. 
X is defined (at all points where the derivative exists) by 


fix (0) = <Fx(2) (1.34) 


Remark. The function fx (x) is not the probability P[X = zx] for a continuous 
r.v. since P|X = x] = 0 Y z in this case. The interpretation that can be given 
to fx (x) is the following: 

Pilr-§<X<a2+ 


€ 


wfm 
lan 


Jx) = (1.35) 


where € > 0. The equality is obtained by taking the limit as e | 0. 


Properties. i) fx(x) > 0 [by the formula (1.35), or by the formula (1.34), 
because Fx is a nondecreasing function]. 
ii) We deduce from the formula (1.34) that 


F(a) =f fx(tyae (1.36) 
It follows that 7 
a fx(x) dz =1 (1.37) 
We also have 7 
Pla < X < b] = Fx(b) — Fx (a) = [ fx (x) dx (1.38) 


Thus, the probability that X takes a value in the interval (a, b] is given by the 
area under the curve y = fx(a) from a to b. 
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Definition 1.2.8. The conditional density function of the continuous r.v. 
X, given an event A for which P[A] > 0, is given by 


fx(x| A) = =F x(a | A) (1.39) 


Remark. We find that the function fx (az | A) may be expressed as follows: 


fx (2) 
= 1.40 
for all x € Sx for which the event A occurs. For example, if Sx = [0,1] and 
A = {X < 1/2}, then Sy = [0, 1/2), where Y := X | A. 


Remark. If X is an r.v. that can take an uncountably infinite number of 
values, but the function Fy is not continuous, then X is an r.v. of mized type. 
An example of an r.v. of mixed type is the quantity X (in inches) of rain 
or snow that will fall during a certain day in a given region. We certainly 
have the following: P[X = 0] > 0, so that X is not a continuous r.v. (since 
P|X = 0] = 0 for any continuous r.v.). It is not an r.v. of discrete type either, 
because it can (theoretically) take any value in the interval [0, o0). 





Example 1.2.5. Suppose that 


1/4 if-l1<az<1 
Ixia) =< 12s) i tae 
0 elsewhere 


We can check that the function fy is nonnegative and that its integral over 
R is indeed equal to 1. 
We calculate 


0 if gz oe | 
(c+1)/4 if-lL<a<l 
(l+Inz)/2ifl<ar<e 

1 iiz>e 


Fx(z)= 


Note that the density function fyx(x) is discontinuous at z = 1 (and at 
x = —1 and x = e), which is allowed, whereas the distribution function Fy is 
a continuous function, as it should be (for a continuous random variable). 

Next, we can calculate Fx(x | X < 0) and differentiate this function 
to obtain fx(z | X < 0). It is, however, more efficient to simply calculate 
P|X < 0] = 1/4 and write that 


lif-l<zx<0 
0 elsewhere 


fx(a|X<0)=] 


Finally, note that the function fx(x | X < 0) also satisfies the two properties 
of probability density functions: it is nonnegative and its integral over R is 
equal to 1. 
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Important continuous random variables 


i) Uniform distribution on the interval [a,b]: 
fx(z)=(b—a)? fora<a<b (1.41) 
Notation: X ~ Ufa, b]. 
ii) Exponential distribution with parameter À > 0: 
fx(x) = Ae** for x >0 (1.42) 
Notation: X ~ Exp(A). 
iii) Gamma distribution with parameters a > 0 and À > 0: 


Ne (Na Se 


Ta) for x > 0 (1.43) 


fx(x) = 


where I"(a) = (a — 1)! if a € N (see p. 115). Notation: X ~ G(a, A). 


iv) Gaussian’ distribution with parameters u and o°, where o > 0: 


fx(x) = 





1 CE) 
exp <4 —=— for x € R 1.44 
Ta P l 553 (1.44) 
Notation: X ~ N(u, a°). 
Remarks. i) In the particular case where u = 0 and o = 1 (see Fig. 1.4), X is 
called a standard Gaussian distribution. Its distribution function is denoted 
by @: 


D(z) 3= / eee dx: (1.45) 


eine: N 2T 


The values of this function are presented in Table A.3, p. 370, for z > 0. By 
symmetry, we may write that (—z) = 1 — &(z). 

ii) If we define Y = aX + b, where X has a Gaussian distribution with pa- 
rameters u and o”, then we find that Y ~ N(ap + b,a?o7). In particular, 
Z := (X — p)/a ~ N(0,1). 


Transformations. If X is a r.v., then any transformation Y := g(X}), where 
g is a real-valued function defined on R, is also a random variable. 


T Carl Friedrich Gauss, 1777-1855, was born and died in Germany. He carried out 
numerous works in astronomy and physics, in addition to his important mathe- 
matical discoveries. He was interested, in particular, in algebra and geometry. He 
introduced the law of errors, that now bears his name, as a model for the errors 
in astronomical observations. 
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1 i x 


Fig. 1.4. Standard Gaussian distribution. 


Proposition 1.2.2. Suppose that the transformation y = g(x) is bijective. 
Then the density function of Y := g(X) is given by 


fy(y) = fx (97*(y)) Fo) (1.46) 


for y € [g(a), g(b)| (respectively, [g(b), g(a)|) if g is a strictly increasing (resp., 
decreasing) function and Sx = [a,b]. 


Example 1.2.6. If X ~ U(0,1) and we define Y = e*, then we obtain 


——  —_ 
— 





bocio $ ii 


for y € (1,e). 


Definition 1.2.9. The mathematical expectation (or the mean) E|X] of 
the r.v. X is defined by 


E[X] = a px(xzk) (discrete case) (1.47) 
k=1 
or 
E| X] = f z fx(x)dxr (continuous case) (1.48) 


Remarks. i) Generalization: we obtain the conditional (mathematical) ex- 
pectation E|X | A] of X, given an event A, by replacing px (x) by px(z | A) 
or fx(x) by fx(x | A) in the definition. 


ii) The mathematical operator E is linear. 


Proposition 1.2.3. The mathematical expectation of g(X) is given by 
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Elg(X)] = X` g(tn)px(xe) (discrete case) (1.49) 
k=1 
Elg(X)| = D g(x) fx(x)dx (continuous case) (1.50) 


Remark. We can calculate the mathematical expectation of g(X) by condi- 
tioning, as follows: 


Elg(X)] = $ Elg(X) | B:]}P[B4 (1.51) 
i=1 
where B,,..., Bn is a partition of a sample space S. 


The next two definitions are particular cases of mathematical expectations 
of transformations g(X) of the r.v. X. 


Definition 1.2.10. The kth moment (or moment of order k) of the r.v. 
X about the origin is given by E[X*], for k =0,1,2,.... 


Definition 1.2.11. The variance of the r.v. X is the nonnegative quantity 
V[X] = B[(X — BX) (1.52) 


Remarks. i) The standard deviation of X is defined by STD[X] = (V[X])1/?. 
The r.v. X and STD[X] have the same units of measure. 


ii) We can also calculate the variance of X by conditioning with respect to a 
partition of a sample space S, together with the formula (1.52): 


VIX] = JD BUX - E[X))? | BIPIBI 


iii) Generalization: the conditional variance of X, given an event A, is de- 


fined by 
VIX | A] = El(X - E|X | Al)? | A] (1.53) 


Proposition 1.2.4. i) V[iaX +b) =a?V[X] Va,beER. 
ii) V[X] = E[X?] — (E[X))?. 


Example 1.2.7. The mean (or the expected value) of the r.v. Y in Example 
1.2.6 is given by 


al 
EY|= | i TSRS] 


We also have 
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Tad y2 |" e-1 
Ely? =| 2-dy= >| = 
LY") eg BI, ; 
It follows that 
— 4e- 3 
v[Y] = (e —1) = L CY ~ 0.2420 
Now, let Z := In Y. We have 

ei _ (ny)? |’ 1 
E|Z| =j myz dy 3 |, 23 





Note that Z is identical to the r.v. X in Example 1.2.6 and that the mean of 
X ~ U(0,1) is indeed equal to 1/2. 


A very important special case of the mathematical expectation E [g(X)] 
occurs when g(X) = e##*, 


Definition 1.2.12. The function 
Cx (w) = Efe??*] (1.54) 
where j = /—1, is called the characteristic function of the r.v. X. 


If X is a continuous r.v., then Cx(w) is the Fourier® transform of the 
density function fx (x): 


= | 
Cx(w)= | e fx(z)dz (1.55) 
We can invert this Fourier transform and write that 
1 aS . 
fx(x) = zl e~e Cy (w) dw (1.56) 
PE E 


Since the Fourier transform is unique, the function Cx (w) characterizes en- 
tirely the r.v. X. For instance, there is only the standard Gaussian distribution 
that possesses the characteristic function Cx (w) = e~” a 


= We can also use the function Cx(w) to obtain the moments of order n 
of the r.v. X, generally more easily than from the definition of E[X™] (for 
n € {2,3,...)). 


Proposition 1.2.5. If the mathematical expectation E|X"] exists and is fi- 
nite for all n € {1,2,...}, then 
E|X"|= (-j 


An ~ Cx (w)| 0 (1.57) 


8 Joseph (Baron) Fourier, 1768-1830, was born and died in France. He taught at 
the Collége de France and at the Ecole Polytechnique. In his main work, the 
Théorie Analytique de la Chaleur, published in 1822, he made wide use of the 
series that now bears his name, but that he did not invent. 
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Table 1.1. Means, Variances, and Characteristic Functions of the Main Random 
Variables (with q := 1 — p) 


Distribution Characteristic function 


Bernoulli pel” +q 
B(n, p) (pett +q)” 


pe?” 


G See ee. a ee | 
eom(p) 1 — qei” 


Poi(A) exp{A(e?” — 1)} 
eiwb a eiva 


a Feb a) 


Exp(A 
xp(A) or 


G(a, A) x2 ( 4 l 


À — Jw 


N(x, 0°) exp{jwu — 5w°o"} 





Remark. Table 1.1 gives the mean, the variance, and the characteristic function 
of all the discrete and continuous random variables mentioned previously. 


Many authors prefer to work with the following function, which, as its 
name indicates, also enables us to calculate the moments of a random variable. 


Definition 1.2.13. The moment-generating function of the r.v. X is de- 
fined, if the mathematical expectation exists, by Mx (t) = Efe'*]. 


Remarks. i) When X is a continuous and nonnegative r.v., Mx(t) is the 
Laplace? transform of the function fx (x). 


ii) Corresponding to the formula (1.57), we find that 


d” 
EIX T= ET Mx(t)|,-9 for n= 1,2,... (1.58) 


? Pierre Simon (Marquis de) Laplace, 1749-1827, was born and died in France. In 
addition to being a mathematician and an astronomer, he was also a minister and 
a senator. He was made a count by Napoléon and marquis by Louis XVIII. He par- 
ticipated in the organization of the Ecole Polytechnique of Paris. His main works 
were on astronomy and on the calculus of probabilities: the Traité de Mécanique 
Céleste, published in five volumes, from 1799, and the Théorie Analytique des 
Probabilités, whose first edition appeared in 1812. Many mathematical formulas 
bear his name. 
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Example 1.2.8. If X ~ Poi(A), we calculate 





We deduce from this formula and from Eq. (1.58) that 
E[X] = My(0)=A and E[X?] =A? +A 


so that V[X] = A +A- (A)? =). 
Note that to obtain E[X?], we can proceed as follows: 


Bhi 2 A^ _ oA a eA eee 
tie DE : i2, d\ (k— 1)! 


dma A d , 
Reed Quai Sage —_(\e z A? 
A a A 


It is clear that it is easier to differentiate twice the function e~* exp(e’A) than 
to evaluate the infinite sum above. 


When we do not know the distribution of the r.v. X, we can use the 
following inequalities to obtain bounds for the probability of certain events. 


Proposition 1.2.6. a) (Markov’s!® inequality) If X is an r.v. that takes 
only nonnegative values, then 


P|X >d< = Ve>0 (1.59) 
b) (Chebyshev’s!! inequality) Jf E[Y] and V[Y] exist, then we have 


Ply — EIY]| > q < a Ve>0 (1.60) 


10 Andrei Andreyevich Markov, 1856-1922, who was born and died in Russia, was a 
professor at St. Petersburg University. His first works were on number theory and 
mathematical analysis. He proved the central limit theorem under quite general 
conditions. His study of what is now called Markov chains initiated the theory of 
stochastic processes. He was also interested in poetry. 

H Pafnuty Lvovich Chebyshev, 1821-1894, was born and died in Russia. By using 
the inequality that bears his name, he gave a simple proof of the law of large 
numbers. He also worked intensively on the central limit theorem. 
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1.3 Random vectors 


Definition 1.3.1. An n-dimensional random vector is a function X = 
(X1,...,Xn) that associates a vector (X1(s),...,Xn(s)) of real numbers with 
each element s of a sample space S of a random experiment E. Each compo- 
nent Xy of the vector is a random variable. We denote by Sx (C R”) the set 
of all possible values of X. 


Remark. As in the case of the random variables, we will use the abbreviation 
r.v., for random vector, since there is no risk of confusion between a random 
variable and a random vector. 


Two-dimensional random vectors 


Definition 1.3.2. The joint distribution function of the r.v. (X,Y) is 
defined, for all points (x,y) € R?, by 


Fy,y(t,y) = PHX <2}n{Y sy} = P[X < z,Y <y] (1.61) 
Properties. i) Fy,y(—0o0, y) = Fy, y(x, —00) = 0 and Fy, y (oo, 00) = 1. 


li) Fx y (21,91) < Fx,y (£2, yo) if z1 < £2 and yı < y2- 
iii) We have 


LNS +E, y) = lim Fx,y (@,y + €) = Fyx,y (z, y) (1.62) 


Remark. We can show that 


Pla< X <b,c<Y <d] = Fy y (b, d) — Fy,y (b,c) — Fy,y (a, d) + Fx,y (a,c) 
(1.63) 


where a,b,c, and d are constants. 


It is easy to obtain the marginal distribution function of X when the 
function Fx y is known. Indeed, we may write that 


Fy (zg) = PX <2 Y < col = Frrr) (1.64) 


Definition 1.3.3. A two-dimensional r.v., Z = (X,Y), is of discrete type 
if Sz is a finite or countably infinite set of points in R°: 


Sz, = Syxy = (Gar re =e eee ‘k= eee (1.65) 


Definition 1.3.4. The joint probability mass function of the discrete r.v. 
(X,Y) is defined by 


px,y (Tj, Yk) = PIX = Tj, Y = Yk] (1.66) 
forj,k= Ey QD oepa 
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To obtain the marginal probability mass function of X, from px y, we 
make use of the total probability rule: 


px (aj) = PIX = zj] = $ PHX = as} N {Y = yeh] = $ px (oj yk) 
k=1 k=1 
(1.67) 


Example 1.3.1. The generalization of the binomial distribution to the two- 
dimensional case is the joint probability mass function given by 


m! 


PX4,X2(£1, £2) = — x2)! pi pz (1 — pi — p2)” “2? (1.68) 


x1!aq!(m — z1 


where z1, £2 € {0,1,... ,m} and zı +22 < m € N. We say that (X1, X2) has 
a trinomial distribution with parameters m, p1, and po, where 0 < pk < 1, for 
k = 1,2. We can generalize further the binomial distribution and obtain the 
multinomial distribution (in n dimensions). 


Definition 1.3.5. A two-dimensional r.v., Z = (X,Y), is of continuous 
type if Sz is an uncountably infinite subset of R?. (We assume that X and 
Y are two continuous random variables. ) 


Remark. We will not consider in this book the case of random vectors with at 
least one component being a random variable of mixed type. 


Definition 1.3.6. The joint (probability) density function of the con- 
tinuous r.v. Z= (X,Y) is defined by 


32 
E 1.69) 
fxy(z,y) Jrg YY) ( ) 


for any point where the derivative exists. 


Remarks. i) Corresponding to the formula (1.67), the marginal (probabil- 
ity) density function of X can be obtained as follows: 


OO 
fx(x) = / fx,y (x,y) dy (1.70) 
-00 
where we integrate in practice over all the values that Y can take when X = zx. 


ii) The probability of the event {Z € A}, where A C Sz, can be calculated as 
follows: 


Pizeaj= f | fxy(@ y) dedy (1.71) 
A 


iii) The distribution function of the continuous r.v. (X,Y) is given by 
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Fy y (x,y) = [. fx, y(u, v) dudv (1.72) 
In the discrete case, the formula above becomes 
Fyy(2,y)= X > px,y(2;, yx) (1.73) 
Ti <2 Uk <y 


Example 1.3.2. The continuous r.v. (X,Y) has the joint density function 


ne ea eae. O<y<e 
fx y(z,y) = 3 


0 elsewhere 


We calculate i 
fx(a)= f — dy = Ing ifl<a<e 


[+= Ine = 


and 
€e 





=> if0<y<1l 











a e 2 € oe oy, 
fr(y) = | Zi ey eae 
y T 2 i 2 
0 elsewhere 


Example 1.3.3. Let 


_ f2ye*ife>0,0<y<1 
fxy(Z,y) = l 0 elsewhere 


First, we calculate 


1 
a) = / 2ye *dy=e* if >0 
0 
and Me 
fr(v) = | 2ye "*dx=2y if0<y<l 
0 
Moreover, we find that 


0 ifx<Oory<0 
Fy y(a,y) = ¢ (l—e7*)y? ifa >0 and0<y<1 
e* ifz>Oandy>1 
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Finally, we calculate 
1 i—y 
PIX +Y >aj=1-Pix+¥sij=1-f f 2y e`” dxzdy 
0 Jo 


1 
=1- | 2y (1 — e7!) dy = 1 — (1 —2e7!) = 2e7! 
0 


because 5 
f ye” dy = yelp — f eVdy=e-—(e-1)=1 
0 0 


Definition 1.3.7. Let (X,Y) be a random vector. We say that X and Y are 
independent random variables if 


Px,y (£j Yk) = px (zipy (ye) if (X,Y) is discrete (1.74) 


or 
fxy.y(z,y) = fx(x)fy(y) if (X,Y) is continuous (1.75) 
Remarks. i) More generally, X and Y are independent if (and only if) 
PIX € A,Y € B] = P|X € AJP[Y € B] (1.76) 


where A (respectively, B) is any event that involves only X (resp., Y). In 
particular, we must have 


Fy y(z,y) = PIX <2,Y < y] = Fy(z)Fy(y) VY (z,y) (1.77) 


ii) If X and Y are independent r.v.s, then so are g(X) and A(Y). 


iii) Let Z := X +Y, where X and Y are two independent r.v.s. Then [see the 
formula (1.102), 


Mz(t) = Ele’@| = Ele**)] = Efe*|EletY] = Mx (t)My(t) (1.78) 
Similarly, Cz(w) = Cx (w)Cy (w). 


Example 1.3.4. We deduce from Eq. (1.75) that the r.v.s X and Y in Example 
1.3.2 are not independent, whereas those in Example 1.3.3 are. 


Definition 1.3.8. If Y is a discrete r.v., then the conditional distribution 
function of X, given that Y = yp, is defined by 


PIX < x, Y = y;] 


PY ii YPY =y] 0 (1.79) 


Fyjy (x | yk) = 


Remark. In theory, Y can be a continuous r.v. in the preceding definition. 
However, in practice, most of the time the two random variables in the pair 
(X,Y) are of the same type. 
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Definition 1.3.9. If (X,Y) is a discrete r.v., then the conditional proba- 
bility mass function of X, given that Y = yp, is defined by 
_ pxyy(%j,yr) _ PIX =2j3,Y = yr] 


pxiy (zj | Yk) = a a if P[Y = yx] > 0 
(1.80) 


Remark. The conditional functions possess the same properties as the corre- 
sponding marginal functions. 


When Y is a continuous r.v., we cannot condition on the event {Y = y} 
directly, because P[Y = y| = 0 for all y. We must rather take the limit as 
dy decreases to zero of the functions defined by conditioning on the event 
{y<Y <y+dy}. We then obtain the following proposition. 


Proposition 1.3.1. If (X,Y) is a continuous r.v. and fy(y) > 0, then the 
conditional distribution function and the conditional density function 
of X, given that Y = y, are given, respectively, by 


an fx.y(u, y) du 
fy (u) 


ond fry(e|y)= O e (81 


Fyyy (x | y) = fy(y) 


Example 1.3.5. The conditional density function of Y, given that X = zx, in 
Example 1.3.2 is 


fx.v(z, y) 1 ‘ 
r) = ———— = - f0<y<e 
That is, Y | {X = x} has a uniform distribution on the interval (0, x). Hence, 
we easily find that Fy)x(y| az) =0 if y <0, 


Y g 
Fyx|z)=7 fO<y<zT 


and Fy|x(y|a2)=1lify> cz. 


Proposition 1.3.2. The r.v.s X and Y are independent if and only if the 
conditional distribution function, the conditional probability mass function, 
or the conditional density function of X, given that Y = y, is identical to the 
corresponding marginal function. 


Example 1.3.6. We say that the continuous random variables X and Y have 
a binormal (or bivariate normal) distribution with parameters ux €E R, 
ly €E R, oe > 0, of, > 0, and p € (—1,1), and we write that (X,Y) ~ 
N(ux, ty; o%, 0%; p), if their joint density function is 


1 


fx,y(t,y) = ea pe 
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1 r-aux\ (y-py\* 
al- ( OX +( oy 


= pp Endu | (1.82) 


OXOY 





for (x,y) € R?. 
We easily find that X ~ N(u.x,0%) and Y ~ N(uy, 0%). It follows that 


X | {Y =y} ~ N(ux + plox/oy)y — uyr) 0% (1 — p*)) (1.83) 


Since X and X | {Y = y} have the same distribution if p = 0, we can state 
that X and Y are independent r.v.s if the parameter p is equal to zero. This 
parameter is actually the correlation coefficient of X and Y. 


Definition 1.3.10. The conditional expectation of X, given that Y = y, 
as defined by 


EX ir =yl= I pxjy (zj |y) (discrete case) (1.84) 
j=l 
or 
OO 
E|X | Y = y] = J x fxjyy(x | y)dx (continuous case) (1.85) 


The mean E|g(X)] of a transformation g of a random variable X is a real 
constant, while Efg(X) | Y = y] is a function of y, where y is a particular 
value taken by the r.v. Y. We now consider Efg(X) | Y]. It is a function of 
the r.v. Y that takes the value E|g(X) | Y = y} when Y = y. Consequently, 
E\g(X) | Y] is a random variable, whose mean can be calculated. We then 
obtain the following important proposition. 


Proposition 1.3.3. We have 


Elg(X)] = E[Elg(X) | Y|] (1.86) 


Remarks. i) We deduce from the preceding proposition that 


i E|X | Y = yg! py (yz) (discrete case) 
E|X| = E[E[X | Y]] = n 
Jo EIX |Y = y] fy (y)dy (continuous case) 
(1.87) 


ii) We can calculate the variance of X by conditioning on another r.v. Y as 
follows: 
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V(X] = E[E[X? | Y]] — (E[E[X | YJ)? (1.88) 

iii) Let X1, Xo,... be r.v.s that possess the same distribution as the r.v. X, 
so that E[X,] = E[X] and V[X;] = V(X], for k = 1,2,..., and let N be an 


r.v. independent of the X;’s and taking its values in the set {1,2,...}. By 
making use of the formula (1.86), we can show (see p. 254) that 


N 
YX 
k=1 


If the r.v.s X, are independent among themselves, we also have (see p. 254) 


N 
2 Xk 
k=1 


E = E[N]E[X] (1.89) 








V = E(N|V[X] + V[N](E[X])* (1.90) 








iv) Suppose that we wish to estimate a random variable X by using another 
r.v. Y. It can be shown that the function g(Y ) that minimizes the mean-square 
error (MSE) 


. MSE := E[(X — g(Y))?| (1.91) 


is 9(Y) = E|X | Y]. If we look for a function of the form g(Y) = aY + 8, we 
can show that the constants a and 8 that minimize MSE are 


E(XY] — ELX]EIY) 


VY and ĝ= E[X]—4E[Y} (1.92) 


Aes 
Finally, if g(Y) = c, we easily find that the constant c that yields the smallest 
MSE is c = EX]. 


The function g(Y) = E[X | Y] is the best estimator of X, in terms of Y, 
while g(Y) = âY + B is the best linear estimator of X, in terms of Y. If X 
and Y both have a Gaussian distribution, then the two estimators are equal 
(see Ex. 1.3.6). 


Proposition 1.3.3 also enables us to calculate the probability of the event 
{X € A} by conditioning on the possible values of an r.v. Y. We only have 
to define the r.v. W such that W = 1 if X € A and W =Oif X ¢ A, and use 
the fact that E[W] = PIX € A]. We can then show the following proposition, 
which is the equivalent of the total probability rule for random variables. This 
proposition and Proposition 1.3.3 will be very useful in the next chapters. 


Proposition 1.3.4. We may write that 


Xa PIX € A| Y = yg] py (yg) (discrete case) 
P(X € A] = (1.93) 
feo PIXEA]|Y =y] fy(y) dy (continuous case) 
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Corresponding to the definition of the conditional variance V[X | A] that 
was given in the preceding section, we now have the following definition. 


Definition 1.3.11. The conditional variance of X, given the r.v. Y, 1s 
defined by 


VIX | Y] = E(X - E[X | Y)? | Y] (1.94) 
Remarks. i) We find that 

VIX | Y] = EIX? | Y] - (EIX | YI) (1.95) 
ii) We can show the following useful result: 

VIX] = E[V|X | Y] + V[E[X |Y] (1.96) 
Erample 1.3.7. Instead of calculating the variance of the r.v. Y in Example 
1.3.2 from the density function fy that we obtained and from the definition of 


V{[Y], we can use the fact (see Ex. 1.3.5) that Y | X ~ U(0, X). It follows that 
E\Y | X] = X/2 and V[Y | X| = iin and then, by the formula (1.96), 


V[Y] = EXI vix] = 1 gx?) — 5(E[X)) ~ 0.4252 
because 


e e 1 
Ex = | z Ingde = Ž(e? +1) and Bx?) = | g Inada = 5 (2e* +1) 


Remark. We can check that 


EY] = (2 +1) ind EY?) = 5 (2 +1) 


Remark. We can consider conditional expectations E[X | AJ, or conditional 
density functions fx y(z,y | A), etc., with respect to more general events 
A, like Y < y, Y > 0, etc., and also with respect to events A that involve 
both random variables, X and Y. For instance, let X; and X; be independent 
random variables having a U(0,1) distribution. We have 


PX x£ ti 1X4 < Xo] 


PIX, < x1, Xı < Xə] A | 
PIX: < Xə] 2 [Xi = fi, ? ” | 2) A 


: T1 1 
ie: f PX; < x1, Xı < z2] dx at rade, + | m dra} 
0 X1 


= x4(2 = zı) 


— 
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so that 
fx, (zı | X1 < X2) =2(1-—21) for0<2,<1 


Next, we have 


1 
1 
E|X: | Xı < Xo] = Tı .2(1 — xı) dz, = 3 
0 
Actually, if we are only looking for the mean of X1, given that Xı < X2, we 
can directly write that 


1 
E[X; | Xı < Xe] = i [ ti T 3 


where we used the formula 


fxi. x(£1, T2) 


for0 < zı <4. <1 
Pe a. Brave 


Jx lut |X Xa) = 


The following proposition is the two-dimensional version of Proposition 
12.2. 


Proposition 1.3.5. Let W := gı(X, Y) and Z := g2(X,Y), where X and Y 
are two continuous r.v.s. If 
1) the system w = gı(x, y), z = ga(x,y) has the unique solution x = hı(w, z), 
y = ha(w, z) 
and 
2) the functions gı and gz have continuous partial derivatives V (x,y) and the 
Jacobian J(x,y) of the transformation is such that 

O91 Og2 g2 gı 


` y) := £91992 _ 092 OJ r, 1.97 


then 
fw.z (w, z) = fx y (hiw, z), how, z)) [J (hi(w, 2), ho(w, z) (1.98) 


Remarks. i) The proposition can be easily generalized to the n-dimensional 
case, where n E€ {3,4,...}. 

ii) In the particular case where X and Y are independent and Z := X +Y, we 
could use the proposition to obtain the density function of Z. We must first 
define an appropriate auxiliary variable W, then calculate the joint density 
function of the r.v. (W, Z), and finally integrate this joint density function 
with respect to w to obtain fz(z). We can also proceed as follows: 


Flej = J. L-1 u)fy(v)dv du => fz(2) = a aaa a 
1 


Note that the density function of Z is the convolution product of the density 
functions of X and Y. 
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Proposition 1.3.6. The mathematical expectation of the random variable 
Z := g9(X, Y ) is given by 
Soja Doha (Tj, Yk) Px,y (Lj, Yk) (discrete case) 
E[Z] = (1.100) 
| SZ o(x,y) fx,y (x,y) dz dy (continuous case) 


Remark. If the mathematical expectations E[X] and E[Y] exist, we have 
ElaX + bY] =aE|X]+bE|Y] Va,beR (1.101) 
Moreover, if X and Y are independent r.v.s and g(X,Y) = gi(X)go(Y), then 
E\g(X,Y)] = Elgi(X)|Elga(¥)] (1.102) 
Definition 1.3.12. The covariance of X and Y is defined by 


Cov[X,Y] = El(X — E[X])(Y — EIY])| = E[XY]—- E[X]E[Y] (1.103) 


Remarks. i) The covariance generalizes the variance, since Cov[X, X] = V[X], 


but the covariance Cov[ X,Y] can be negative. For example, if Y = —X, then 
we have 
Cov[X, Y] = Cov[X, —X] = E[X(—X)] — E[X|E[-X] = -V[X] < 0 
(1.104) 


ii) We deduce from Eq. (1.102) that if X and Y are independent, then 
Cov[X, Y] = 0. However, the converse is not always true. 


iii) We also define the correlation coefficient of X and Y by 


Cov[X, Y] 


ie ee 1.105 
PXY = STD[X|STD[Y] ee 


We then deduce from Example 1.3.6 that, in the case of the bivariate normal 
distribution, the r.v.s X and Y are independent if and only if their correlation 
coefficient is equal to zero. 


An important particular case of transformations of random vectors is the 
one where the random variable Z := g(X1,... , Xn) is a linear combination of 
the r.v.s Xi, Sais oP, 


Z = ao +a,X1 +--+ + anXn (1.106) 


where the a,’s are real constants V k. We can show the following proposition. 
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Proposition 1.3.7. Let Z be a linear combination of the r.v.s X1,...,Xn- 
We can write (if the mathematical expectations exist) that 


E\Z| = ag + ay EX] +--+ + an Fl X,| (1.107) 
and 
VIZ] =Y aRV[X4] +20 Y ajax Cov[Xy, Xi] (1.108) 
k=1 j=1 k=1 
a ee 
j<k 


Example 1.3.8. If X and Y are two independent r.v.s having a uniform distri- 
bution on the interval [0,1] and Z := X +Y, then Sz = [0,2] and 


oO 1 
ra= fx (u) fy (z—w)du = | fy (2 -—u) du 


Since 


lifz—-i<u<z 
0 elsewhere 


fr(z-w =f 
we may write that 


zZ 
/ 1ay=]2> ALU ee 1 
0 


_ i 
fz (z) = / ldu=2-—zifl<z<2 
z—1 


0 elsewhere 


Remark. If we define the auxiliary variable W = X, then we find that 
fw z (w,z)=1 ifO0<w<10<z2<2,w<z<wtl 


Integrating fw.z(w,z) with respect to w, we retrieve the function fz (2) 
above. 


Next, if Z := X — 5 and W := Z?, we calculate 
Cov[Z, W] = E[ZW] — E[Z|E[W] = E[Z*] -0 =0 


because E[Z?*t"] = 0, for all k € {0,1,...}. However, Z and W are not 
independent, since Z = 0 > W = 0, in particular. 
Finally, Eq. (1.108) enables us to write that 


VÍZ —3W] = VÍZ] + 9V[W] — 6Cov[Z,W] = V x = J +9V 
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1 1 
= VIX] +9V[T?], where T~ U l-33! 


2 2 
1 1 2 
-5+5 


because 


1/2 1/2 2 
V(T?} = EIT$] - (E(T’))? = J t*-1dt— J t. ldt 
-1/2 -1/2 
ee eee ee 2“ _ 1 
5x16 3x4/ 80x144 180 


We continue with limit theorems that will be used in the subsequent chap- 
ters. 


Proposition 1.3.8. Let Xı,X2,... be an infinite sequence of independent 
and identically distributed (i.i.d.) r.v.s, and let Sn := Xi +--+ Xn. 


a) (Weak law of large numbers) Jf E[X1] = u E R, then 


lim P| a <e =1. Ve>0 (1.109) 


NCO 


b) (Strong law of large numbers) If E[X?] < co, then we may write that 


P | i te u! =1 (1.110) 


n= TL 


c) (Central limit theorem) If E[X,] = u € R and V|X;] = o? € (0,00), 
then we have 


j — < z| = PIN (0.1) < 1.111 
sim P| Tie <;| [N (0,1) < 2] ( ) 


Remarks. i) Actually, the condition E[X?] < œ is a sufficient condition for 
the strong law of large numbers to hold. It may be replaced by the weaker 
condition E[|X4|] < oo, which reduces to E[X;}] < œ in the case when X; > 0. 


ii) The central limit theorem (CLT) implies that 
2 Sn 2 
Sn © N(np,no*) and a N(p, 0% /n) (1.112) 


In general, from n = 30, the approximation by the Gaussian distribution 
should be rather good. 
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iii) An application of the CLT is the Moivre!? -Laplace Gaussian approxima- 
tion to a binomial distribution: 


P|B(n,p) = k| = fz(k), where Z ~ N(np, np(1 — p)) (1.113) 


For the approximation to be good, the minimum between np and n(1 — p) 
should be at least equal to 5. 


Example 1.3.9. If Xk ~ Exp(1), for k = 1,... ,30, and if the X,’s are inde- 
pendent r.v.s, then we can show that 


S39 := X1 +... + X39 ~ G(30, 1) 
Making use of the formula 
P[G(n, A) < z] = P[Poi(Az) > n] (1.114) 


we obtain (from a table of the distribution function of the Poisson distribution) 
that 
P{S39 < 30] = P[Poi(30) > 30] = 0.5243 


The approximation by the CLT yields 


P|S30 < 30] = P[N(30, 30) < 30] = 


Example 1.3.10. Suppose that 1% of the tires manufactured by a certain com- 
pany do not conform to the norms (or are defective). What is the probability 
that among 1000 tires, there are exactly 10 that do not conform to the norms? 


Solution. Let X be the number of tires that do not conform to the norms 
among the 1000 tires. If we assume that the tires are independent, then X 
has a binomial distribution with parameters n = 1000 and p = 0.01. We seek 


P|X = 10) ~ fz(10), where Z ~ N(10,9.9) 
ee „f 100- 10)? 
~ Var/o9 C 2 9.9 


Remarks. i) In fact, we obtain that P[X = 10] ~ 0.1257. By using the Poisson 
approximation (see p. 12), we find that 


} œ 0.1268 


12 One of the pioneers of the calculus of probabilities, Abraham de Moivre, 1667- 
1754, was born in France and died in England. The definition of independence of 
two events can be found in his book The Doctrine of Chance published in 1718. 
The formula attributed to Stirling appeared in a book that he published in 1730. 
He later used this formula to prove the Gaussian approximation to the binomial 
distribution. 
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P|X = 10] ~ P[Poi(10) = 10] ~ 0.1251 


In this example, the Poisson approximation is slightly more accurate. However, 
if we increase the value of the probability p, the Moivre-Laplace approxima- 
tion should be better. 


ii) To calculate approximately a probability like P[5 < X < 10], we would 
rather use the distribution function of the Gaussian distribution. It is then 
recommended to make a continuity correction to improve the approximation. 
That is, we write that 


PIB < X < 10} = Pi4.5 < X < 10.5) ~ P[4.5 < Z < 10.5] 


1.4 Exercises 


Section 1.1 


Question no. 1 

In urn A, there are four red balls and two white balls, while urn B contains 
two red balls and four white balls. We throw, only once, a coin for which the 
probability of “tails” is equal to p (0 < p <.1). If we get “tails,” then we will 
draw balls from urn A; otherwise, urn B will be used. 


(a) What is the probability of obtaining a red ball on any draw? 


(b) If we obtained a red ball on each of the first two draws, what is the 
probability of obtaining a red ball on the third draw? 


(c) If we obtained a red ball on each of the first n draws, what is the probability 
that we are using urn A? 


Question no. 2 i 

Box 1 contains 1000 transistors, of which 100 are defective, and box 2 
contains 2000 transistors, of which 100 are also defective. A box is taken 
at random and two transistors are drawn from it, at random and without 
replacement. 


(a) Calculate the probability that both transistors are defective. 


(b) Given that both transistors are defective, what is the probability that they 
come from box 1? 


Question no. 3 

Assume that there is a leap year every four years. How many (independent) 
persons must be in a room, at minimum, if we want the probability that at 
least one of these persons was born on February 29 to be greater than 1/2? 
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Question no. 4 

Object O; moves on the z-axis between 0 and 2, while object O2 moves 
on the y-axis between 0 and 1. Suppose that the position of each object is 
completely random. What is the probability that the distance between the 
two objects is greater than 1? 


Question no. 5 

A certain user of the public transport system can take bus no. N; or bus 
no. Nz to go to work. A bus no. N; runs near his home every hour, from 6:00 
a.m., while a bus no. Nə runs there d minutes after the hour, where d € (0, 30]. 
The user arrives at the bus stop at a completely random time between 7:45 
a.m. and 8:15 a.m. What is the value of d if he takes a bus no. N; thrice more 
often than a bus no. No? 


Question no. 6 

Two sport teams play a series of (independent) games to win a trophy. The 
first team that wins four games gets the trophy. There are no draws. What is 
the probability that a team having, for each game, only a one-in-three chance 
of winning gets the trophy? 


Question no. 7 
In how many ways can we permute the numbers 1,2,...,n if we do not 
want a single number to remain in its original position? 


Question no. 8 

In the dice game called craps, the player tosses two (fair) dice simultane- 
ously. If the sum of the two numbers that show up is equal to 7 or 11, the 
player wins. If the sum is equal to 2, 3, or 12, he loses. When the sum is a 
number z different from the preceding numbers, the player must toss the two 
dice again until he gets a sum equal to z or 7. If x is obtained first, the player 
wins; otherwise, he loses. What is the probability that the player wins? 


Question no. 9 

A says that B told her that C has lied. If the three persons tell the truth 
and lie with probability p € (0,1), independently from one another, what is 
the probability that C has indeed lied? 


Question no. 10 

A man takes part in a television game show. At the end, he is presented 
with three doors and is asked to choose one among them. The grand prize is 
hidden, at random, behind one of the doors, while there is nothing behind the 
other two doors. The game show host knows where the grand prize has been 
hidden. Suppose that the man has chosen door no. 1 and that the host tells 
him that he did well in not choosing door no. 3, because there was nothing 
behind it. He then offers the man the opportunity to change his choice and, 
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therefore, to select door no. 2 instead. What is the probability that the man 
will win the grand prize if he decides to stick with door no. 1? 


Section 1.2 


Question no. 11 

Boxes I and IT both contain n transistors. At each step, a fair coin is tossed. 
If “heads” (respectively, “tails”) is obtained, we take, at random and without 
replacement, a transistor in box I (resp., II). We repeat this experiment until 
one of the two boxes is empty. Let N be the number of transistors that remain 
in the other box at that moment. If we assume that the repeated trials are 
independent, what is the probability mass function of N? 


Question no. 12 | 
Let X be a continuous random variable whose density function is given by 


fx(xz) =c’re"™ = for x > 0 


where c is a positive constant. Calculate E[X | X < 1]. 


Question no. 13 

A mathematician hesitates between three methods to solve a certain prob- 
lem. With the first (respectively, second) method he will work in vain for two 
(resp., three) hours, while the third method will give him the solution at 
once. If we assume that at each step the mathematician uses a method taken 
at random among those that he still has not tried, what is the variance of the 
number of hours that he will have to work to solve his problem? 


Question no. 14 
Let X be a random variable whose moment-generating function, Mx (t), 
exists for t € (—c,c). Show that 


P[X >a] <e"“Mx(t) for0<t<c 


and 
P[X <a] <e""Mx(t) for -c<t<0 


where a is a real constant. 


Question no. 15 
Suppose that the moment-generating function of X exists for every real 
value of t and is given by 


erase. 
Mx(t) = 4 gp HEF 
1 ift=0 
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Use the results of the preceding question to show that 


P[X >1)=0 and P[X < —1]=0 


Question no. 16 

Let X be a continuous random variable whose set of possible values is the 
interval [a,b]. We define Y = g(X). 
(a) Calculate the probability density function of Y if g(x) = 1 — Fy (x). 
Indication. The inverse distribution function ES exists. 


(b) Find a transformation g(x) such that 


fru) = 5 forl<y<3 


Question no. 17 
Calculate EX | X > 1], where X is a random veel having a standard 
Gaussian distribution. 


Question no. 18 
Two players, X and Y, take turns at tossing a fair coin. The first one that 
gets “tails” wins. Calculate, assuming that X starts, 


(a) the probability that X wins, 


(b) the probability that X wins, given that she did not obtain “tails” on her 
first two trials, 


(c) the average number of tosses feeds to end the game, given that X lost. 


Question no. 19 
Suppose that the probability that a family has exactly n children is given 
by 
Pen Sora. 


where c > 0 and 0 < p < 1, and P = 1 — D Pa. Suppose also that every 
child is equally likely to be a male or a female. 


(a) Calculate the probability that a family with n children has exactly k male 
children, for k = 0, 1,...,n 


(b) Find the probability that a family has no male children. 


(c) What is the average number of male children per family? 


Question no. 20 

A box contains 200 brand A and 10 brand B transistors. Twenty transistors 
are taken at random. Let X be the number of brand A transistors obtained. 
(a) Calculate P[X = 20], assuming that the transistors are taken without 
replacement. 
(b) Calculate P[X = 19] if the transistors are taken with replacement. 
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(c) Use a Poisson distribution to calculate approximately PLX = 18] when 
the transistors are taken with replacement. 


Question no. 21 
The density function of the random variable X is given by 


_ f[k(l-22)if0<2<1 
fx(z) = l 0 elsewhere 


where k is a positive constant. 

(a) Calculate fx (a | X? < 1/4). 

(b) Find the constant b that minimizes E[(X — b)?}. 
(c) Find the constant c that minimizes E[|X — cl]. 


Indication. The value £m for which Fy (£m) = 1/2 is called the median of the 
continuous r.v. X. It can be shown that, for any real constant a, 


Elle — al] = Elle - 2ml] +2 f "(2 - a)fx (2) dz 


Question no. 22 

We say that the continuous random variable X, whose set of possible values 
is the interval [0,00), has a Pareto?’ distribution with parameter 0 > 0 if its 
density function is of the form 


; 
O e 
fx(e) =} UF TZS 


0 elsewhere 


In economics, the Pareto distribution is used to describe the (unequal) dis- 
tribution of wealth. Suppose that, in a given country, the wealth X of an 
individual (in thousands of dollars) has a Pareto distribution with parameter 
Ga R 

(a) Calculate fx(2|1 < X <3). 

(b) What is the median wealth (see Question no. 21) in this country? 


(c) We find that about 11.65% of the population has a personal wealth of at 
least $5000, which is the average wealth in this population. What percentage 
of the total wealth of this country does this 11.65% of the population own? 


Question no. 23 
Let > 
majs “a /2 if >0 
0 elsewhere 


13 Vilfredo Pareto, 1848-1923, born in France and died in Switzerland, was an 
economist and sociologist. He observed that 20% of the Italian population owned 
80% of the wealth of the country, which was generalized by the concept of Pareto 
distribution. 
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where k is a positive constant. 
(a) Calculate fx (a | X < 1.282). 


Indication. We have P|N(0,1) < 1.282] ~ 0.9. 
(b) Find the value of xg for which Fy (xp) ~ 0.35. 


Remark. The random variable X defined above actually has a Maxwell!’ dis- 
tribution with parameter a = 1. In the general case where a > 0, we may 
write that 
\/2 2 
x(x) = V2/T 126-8/(20 ) ifx>0 
a 

We find that E[X] = 2a,/2/m and V[X] = a?[3 — (8/r)|. This distribu- 
tion is used in statistical mechanics, in particular, to describe the velocity of 
molecules in thermal equilibrium. 


Question no. 24 
Let X be a continuous random variable having the density function 


4 ae 2 
2-2? /(26) 


fx(x) = l 02 


if x > 0 


0 elsewhere 


We say that X has a Rayleigh’ distribution with parameter 6 > 0. 

(a) Show that E[X] = 0,/7/2 and V[X] = 6? [2 — (1/2)]. 

(b) Let Y := In X, where X has a Rayleigh distribution with parameter @ = 1. 
Calculate (i) fy (1) and (ii) the moment-generating function of Y at t = 2. 


(c) We define Z = 1/X. Calculate the mathematical expectation of Z if @ = 1 
as in (b). 


Section 1.3 


Question no. 25 

The lifetime X (in days) of a device has an exponential distribution with 
parameter À. Moreover, the fraction of time during which the device is used 
each day has a uniform distribution on the interval [0, 1], independently from 
one day to another. Let N be the number of complete days during which the 
device is in a working state. 


(a) Show that PIN > n} = (1 — e7*)"/\*, for n =1,2,.... 


14 James Clerk Maxwell, 1831-1879, was born in Scotland and died in England. He 
was a physicist and mathematician who worked in the fields of electricity and 
magnetism. 

15 John William Strutt Rayleigh, 1842-1919, was born and died in England. He 
won the Nobel Prize for physics in 1905. The distribution that bears his name 
is associated with the phenomenon known as Rayleigh fading in communication 
theory. 
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Indication. Because of the memoryless property of the exponential distribu- 
tion, that is, — 


PIX >s+t| X >t]=P|X >s] Vs,t2>0 


it is as if we started anew every day. 
(b) Calculate E[N | N < 2] if A = 1/10. 


Question no. 26 
Let X, and X2 be two independent N(u, a?) random variables. We set Y; 
= X4 F Xo and Yo = Xı + 2X9. 
(a) What is the joint density function of Y; and Y2? 
(b) What is the covariance of Y; and Y3? 


Question no. 27 

Let Xı and Xə be two independent random variables. If X, has a gamma 
distribution with parameters n/2 and 1/2, and Y := X; + X2 has a gamma 
distribution with parameters m/2 and 1/2, where m > n, what is the distri- 
bution of X>? 


Indication. If X has a gamma distribution with parameters a and A, then (see 
Table 1.1, p. 19) 
Ae 
C = 
ear E 


Question no. 28 
Show that if E[(X — Y)?] = 0, then P|X = Y] = 1, where X and Y are 
arbitrary random variables. 


Question no. 29 
The conditional variance of X, given the random variable Y, has been 
defined (see p. 28) by 


VIX |Y] = BUX - EIX | Y]}* | Y] 
Prove the formula (1.96): 
V[X] = BIV[X | Y]] + V[E[X | Y] 
Question no. 30 
Let X be a random variable having a Poisson distribution with parameter 


Y, where Y is an exponential r.v. with mean equal to 1. Show that W := X +1 
has a geometric distribution with parameter 1/2. That is, 


pw(n) = (1/2) for n= 1,2,... 


1.4 Exercises Al 


Question no. 31 
Let X; and X> be two independent random variables, both having a stan- 
dard Gaussian distribution. 
(a) Calculate the joint density function of Y; := X? + X? and Yo := Xo. 
(b) What is the marginal density function of Y;? 


Question no. 32 

Let X and Y be independent and identically distributed random variables. 
(a) Show that h(z) := EIX | X +Y =z] = 2/2. 
(b) Evaluate E[(X — h(Z))?) in terms of V[X]. 


Question no. 33 
A company found out that the quantity X of a certain product it sells 
during a given time period has the following conditional density function: 


4 
Ixy (x ly) = poe ne 0 


where Y is a random variable whose reciprocal Z := 1/Y has a gamma dis- 
tribution. That is, 


(Azami 


f >0 
Tla or z > 


fz(z) =r” 
(a) Obtain the marginal density function of X. 
(b) Calculate EJZ | X = a]. 
Question no. 34 
Let X and Y be continuous and independent random variables. Express 


the conditional density function of Z := X + Y, given that X = x, in terms 
of fy. 


Question no. 35 
Show that for continuous random variables X and Y, we have 


BY |X <a] = pry | BY |X =ulfx(u)au 
ERST. 


Question no. 36 
Let X1,... , Xn be independent random variables such that 





= akfa 
ix, (2) = x? +a? 
for —œ < q < œ and k =1,2,...,n, where ak > OV k. 


(a) Calculate the density function of the sum Z := Xi +... + Xn. 
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Indication. The characteristic function of Xķę is given by 
Cx, (w) = e~ ante 


(b) Assuming that a, = a1, for k = 2,3,..., can we state that fz(z) tends 
toward a Gaussian probability density? Justify. 


Question no. 37 

Suppose that X ~ N(0,1) and Y ~ N(1, 1) are random variables such that 
Px.y = p, where px.y is the correlation coefficient of X and Y. Calculate 
E(X?Y?). 


Question no. 38 


Let 
eYifz>O,y>r 
0 elsewhere 


f X,Y (x ’ y) = 
be the joint density function of the random vector (X,Y). 


(a) Find the estimator g(Y) of X, in terms of Y, that minimizes the mean- 
square error MSE := E[(X — g(Y))?]. 


(b) Calculate the minimum mean-square error. 


Question no. 39 
The joint density function of the random vector (X,Y) is given by 


3(¢2 + ay +y?) if -1<x2<land-l<y<1 

— 18 

Ixy (ty) = { 0 elsewhere 

Calculate (a) E[X | Y = 1/2], (b) E[Y | X], (c) the mean-square error MSE 
:= El(Y — g(X))*] made by using g(X) := E[Y | X] to estimate the random 
variable Y. 


Indication. It can be shown that 


MSE = E[Y?| — E[g?(X)|_ if g(X) = E[Y | X] 


Question no. 40 

A number X is taken at random in the interval [0,1], and then a number 
Y is taken at random in the interval [0, X]. Finally, a number Z is taken at 
random in the interval [0, Y]. Calculate (a) E[Z], (b) V[Y], (c) P[Z < 1/2]. 
Question no. 41 | 

An angle A is taken at random in the interval [0, 7/2], so that 


fala) = Z for 0 <a < T/2 
Let X := cos Á and Y := sin A. Calculate 


(a) PIX =1|Y =0], (b) EY |X], (c) E[Y], (d) EIX |X +Y], 
(e) E [X?| X +Y], (f) E[X| A], (g) PIX =0 | {X =0}U {X = v3/2}. 
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Indication. We have 


d 1 
— arccos rt = ———  for—-l<a2<1 


dx AEn 
(h) VIX | Y] if the angle A is taken at random in the interval (0,7). 


Question no. 42 
Let 
fxi (2 |y) =y" forr>0,0<y<1 


Calculate, assuming that Y has a uniform distribution on the interval (0, 1), 
(a) fx(z), (b) PIXY >1], (o) V[X|Y], (d) ELA]. 


Question no. 43 | 

We suppose that the (random) number N of customers that arrive at 
an automatic teller machine to withdraw money, during a given hour, has 
a Poisson distribution with parameter \ = 5. Moreover, the amount X of 
money withdrawn by an arbitrary customer is a discrete random variable 
whose probability mass function is given by 


1 
px(£) = A if z = 20, 40, 60, 80, or 100 


Finally, we assume that N and X are independent random variables. Let Y 
be the total amount of money withdrawn over a one-hour period. Calculate 


(a) EIY | N >00], (b) PIY = 60}, (c) PIN=3]|Y = 60]. 
Question no. 44 

Let Xı and Xz be two independent random variables having a uniform 
distribution on the interval (0,1). We define Y = max{ X1, X2}. Calculate (a) 
Fyix (y | z1) (b) BY | Xı = 21}, (c) V[EIY | Xij), (d) E[VIY | Xi]. 
Indication. If X is a nonnegative continuous (or mixed type) random variable, 
then 


E[X] = I Tenoa 


Question no. 45 

We consider a system made up of two components placed in parallel and 
operating independently. That is, both components operate at the same time, 
but the system functions if at least one of them is operational. Let T; be the 
lifetime of component i, for i = 1,2, and let T be the lifetime of the system. 
We suppose that T; ~ Exp(1/2), for i = 1,2. Calculate 
(a) EITT =1), b ETD >11}, (c) EIT | {r >1}U {n > 1}. 
Indication. In (c), we can use the formula 


E[X | AU B] = E[X | A] P[A| AU B] + E[X | B] P[B | AUB 
— E[X | AN B] PJAN B| AUB] 
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Question no. 46 
Let Xı ~ U[-1, 1], X2 ~ U[0, 1], and X3 ~ U0, 2] be independent random 
variables. Calculate 


(a) P[Xy < Xə), (b) PIX, < Xo < X3], (c) E 
where Y := Xı + Xz and Z := X1 + X3. 





Xı 
h| @ 2PX 


Question no. 47 
Suppose that Xı and Xz are independent random variables such that 


fxi (zi) = }A;e7™lel for z; € R 


where à; is a positive constant, for i = 1,2. Calculate 

(a) PIX: < X2], (b) V[X1 | X1 > 0}, (c) Bl] Xi] | |X] > Y, 
(d) E(x, + Xə | Xı< Xo] if Ay = Ao. 

Question no. 48 


Calculate P[Y > X] if X ~ B(2,1/2) and Y ~ Poi(1) are two independent 
random variables. 


Question no. 49 
Use the central limit theorem to calculate (approximately) 


P[X,+...+ Xo < X41 +... + X100] 


where X1,...,Xio9 are independent random variables, each having a U(0, 1] 
distribution. 


Question no. 50 
Suppose that the random variables X1,..., X39 are independent and all 
have the probability density function 


Bos- forl<a<e 
T 


What is the approximate density function of the product X1 X2: X30? 


Question no. 51 

Let (X,Y) be a random vector having a bivariate normal distribution. 
(a) Calculate P[XY < 0] if ux =0, uy =0, 0% = 1, o} = 4, and p = 0. 
(b) What is the best estimator of X? in terms of Y when px = 0, py = 0, 
o% = 1, oè = 1, and p= 0? 
(c) Calculate E[XY] when py = 1, py = 2, 0% = 1, 0% = 4, and p= 1/2. 


Question no. 52 
Let X and Y be two random variables, and let g and h be real-valued 
functions. Show that 
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(a) Elg(X) | X] = 9(X), 
(b) Elg(X)A(Y)] = E [h(Y)Blg(X) | YJ}. 
Question no. 53 


Letters are generated at random (among the 26 letters of the alphabet) 
until the word “me” has been formed, in this order, with the two most recent 
letters. Let N be the total number of letters that will have to be generated to 
end the random experiment, and let X;, be the kth generated letter. It can be 
shown that E[N] = 676 and V[N] = 454,948. Calculate (a) EIN | X2 = el, 
(b) EIN | X, = mj, and (c) E[N? | Xı = m]. 

Remark. The variable X;, is not a random variable in the strict sense of the 
term, because its possible values are not real numbers. We can say that it is an 
example of a qualitative (rather than quantitative) variable. It could easily be 
transformed into a real random variable by defining X; instead to be equal to 
7 if the kth generated letter is the jth letter of the alphabet, for j = 1,... , 26. 


Question no. 54 
Let X;, for i = 1,2,3, be independent random variables, each having a 
uniform distribution on the interval (0,1). Calculate 
(a) E({X, +Xo+ X3 | Xj + Xo], 
(b) E [X1 + 4X2 | Xi + X2 + X3], 
(c) E\V [XxX | X,+Xet+ X3ii. 
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Stochastic Processes 


2.1 Introduction and definitions 


Definition 2.1.1. Suppose that with each element s of a sample space S of 
some random experiment E, we associate a function X(t,s), where t belongs 
to T C R. The set {X(t,s),t € T} is called a stochastic (or random) 
process. 


Remarks. i) The function X(t, s) is a random variable for any particular value 
oft. 


ii) In this book, the set T will generally be the set N° = {0,1,...} or the 
interval [0, 00). 


Classification of the stochastic processes 


We consider the case when T is either a countably infinite set or an uncount- 
ably infinite set. Moreover, the set of possible values of the random variables 
X(t,s) can be discrete (that is, finite or countably infinite) or continuous 
(that is, uncountably infinite). Consequently, there are four different types of 
stochastic processes (s.p.). 


Definition 2.1.2. [fT is a countably infinite set (respectively, an interval or 
a set of intervals), then {X(t,s),t € T} is said to be a discrete-time (resp., 
continuous-time) stochastic process. 


Remarks. i) Except in Section 2.3, it will not be necessary to write explicitly 
the argument s of the function X(t, s). Thus, the stochastic process will be 
denoted by {X(t),t € T}. However, in the discrete case, it is customary to 
write {Xn n E T}. 

ii) We will not consider in this book the case when T is the union of a set of 
points and of an uncountably infinite set. 
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Fig. 2.1. Example of a random walk. 


Example 2.1.1. A classic example of a stochastic process is the one where we 
consider a particle that, at time 0, is at the origin. At each time unit, a coin 
is tossed. If “tails” (respectively, “heads” ) is obtained, the particle moves one 
unit to the right (resp., left) (see Fig. 2.1). Thus, the random variable X,, 
denotes the position of the particle after n tosses of the coin, and the s.p. 
{Xn,n = 0,1,...} is a particular random walk (see Chapter 3). Note that 
here the index n can simply denote the toss number (or the number of times 
the coin has been tossed) and it is not necessary to introduce the notion of 
time in this example. 


Example 2.1.2. An elementary continuous-time s.p., {X(t),t > 0}, is obtained 
by defining 
XG =7t forts 0 


where Y is a random variable having an arbitrary distribution. 


Definition 2.1.3. The set Sx) of values that the r.v.s X(t) can take ts called 
the state space of the stochastic process {X(t),t € T}. If Sx is) ts finite or 
countably infinite (respectively, uncountably infinite), {X(t),t € T} ts said to 
be a discrete-state (resp., continuous-state) process. 


Example 2.1.3. The random walk in Example 2.1.1 is a discrete-time and 
discrete-state s.p., since Sx, = {0,+1,+2,...}. For the continuous time s.p. 
in Example 2.1.2, it is a continuous-state process, unless Y takes on the value 
0, because Sx) = [0,00) if Y > 0 and Sx) = (—oo, 0] if Y < 0. 


As we mentioned above, for any fixed value of t, we obtain a random 
variable X(t) (= X(t,s)). Although many authors use the notation X (t) to 
designate the stochastic process itself, we prefer to use the notation {X(t),¢ € 
T} to avoid the possible confusion between the s.p. and the random variable. 
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Definition 2.1.4. The distribution function of order k of the stochastic 
process {X(t),t € T} is the joint distribution function of the random vector 
(X(tı), es ,X(tk)): 


E ties: »Lk;5e1,--- stk) = P{X (tı) Sipek X (tk) < Tkl] (2.1) 


Similarly, we define the probability mass and density functions of order 
k of an s.p.: 


DE rise Nin ein gM) = d A = Terra An EE] (2.2) 
and (where the derivative exists) 


ar 
E ee: eee, E T 2.3 
f(x Tk; ti k) e o (Ti oe k) (2.3) 


Remark. When k = 1 or 2, the preceding definitions are in fact only new 
notations for functions already defined in Sections 1.2 and 1.3. 


Example 2.1.4. If the tosses of the coin are independent in Example 2.1.1, 
then we may write, with p := P[{Tails}], that the first-order probability mass 
function (or probability mass function of order 1) of the process at time n = 2 
is given by 


2n(1 — p) ifz =0 
2 . 
ee eos p ice 2 
piajn=2)=PiXg=2)= a gee 
0 otherwise 
First- and second-order moments of stochastic processes 


Just like the means, variances, and covariances enable us to characterize, 
at least partially, random variables and vectors, we can also characterize a 
stochastic process with the help of its moments. 


Definition 2.1.5. The mean E[X(t)] of an s.p. {X(t),t € T} at time t is de- 
noted by mx (t). Moreover, the autocorrelation function and the autoco- 
variance function of the process at the point (t1,t2) are defined, respectively, 
by 


Rx (ti, te) = EIX (t1)X (t2) (2.4) 
and 
Cx (ti, te) = Rx(t1, te) — mx (t1)mx (t2) (2.5) 
Finally, the correlation coefficient of the s.p. at the point (t1,t2) is 


Cx (t1, ta) (2.6) 


A a ae 
px (titz) [Cx (ti, t1) Cx (t2, ta)]!/? 
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Remarks. i) In the case of pairs of random variables, the quantity E[XY| 
is called the correlation of the vector (X,Y). Here, we use the prefix “auto” 
because the function is calculated for two values of the same stochastic process 
{X(t),t € T}. The function Rx y(ty,te) := E[X(ti)Y(te)|, where {Y (t),t € 
T*} is another s.p., is named the cross-correlation function, etc. In fact, we 
could simply use the term correlation function in the case of the function 
Ry (ti, t2). 

ii) The function Rx (t,t) = E[X?(t)| is called the average power of the stochas- 
tic process {X(t),¢ € T}. Furthermore, the variance of the process at time t 
is 


VIX =C Crit) (2.7) 


Since V[X(t)] > 0, we then deduce from Eq. (2.6) that px(t,t) = 1. 


Two properties of stochastic processes that will be assumed to hold true 
in the definition of the Wiener! process (see Chapter 4) and of the Poisson? 
process (see Chapter 5), in particular, are given in the following definitions. 


Definition 2.1.6. If the random variables X(t4) — X(t3) and X(t2) — X (tı) 
are independent V ty < te < t3 < t4, we say that the stochastic process 
{X(t),t € T} is a process with independent increments. 


Definition 2.1.7. If the random variables X(t2+s)— X(ti+s) and X (te) - 
X (tı) have the same distribution function for all s, {X(t),t € T} is said to 
be a process with stationary increments. 


Remarks. i) The random variables X (t +s) — X(t; +s) and X (tz) ~ X(¢1) in 
the preceding definition are identically distributed. However, in general, they 
are not equal. 


ii) The Poisson process is a process that counts the number of events, for 
instance, the arrival of customers or of phone calls, that occurred in the in- 
terval [0,¢]. By assuming that this process possesses these two properties, we 
take for granted that the r.v.s that designate the number of events in disjoint 
intervals are independent, and that the distribution of the number of events 
in a given interval depends only on the length of this interval. In practice, 
we can doubt these assertions. For example, the fact of having had many (or 


! Norbert Wiener, 1894-1964, was born in the United States and died in Sweden. 
He obtained his Ph.D. in philosophy from Harvard University at the age of 18. 
His research subject was mathematical logic. After a stay in Europe to study 
mathematics, he started working at the Massachusetts Institute of Technology, 
where he did some research on Brownian motion. He contributed, in particular, 
to communication theory and to control. In fact, he is the inventor of cybernet- 
ics, which is the “science of communication and control in the animal and the 
machine.” 

? See p. 12. 
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very few) customers on a given morning should give us some indication about 
how the rest of the day will unfold. Similarly, the arrival rate of customers 
at a store is generally not constant in time. There are rush periods and slack 
periods that occur at about the same hours day after day. Nevertheless, these 
simplifying assumptions enable us, for instance, to obtain explicit answers to 
problems in the theory of queues. Without these assumptions, it would be 
very difficult to calculate many quantities of interest exactly. 


Example 2.1.5. Independent trials for which the probability of success is the 
same for each of these trials are called Bernoulli trials. For example, we can 
roll some die independently an indefinite number of times and define a success 
as being the rolling of a “6.” 

A Bernoulli process is a sequence X1, X2,... of Bernoulli r.v.s associated 
with Bernoulli trials. That is, X% = 1 if the kth trial is a success and X, = 0 
otherwise. We easily calculate 


E Xk] =p Vke AATA 
where p is the probability of a success, and 


2 if ki Ak 


It follows that Cx (ky, kg) = 0 if ky A ka and Cx (ky, k2) = p(1 — p) if kı = k2. 
Example 2.1.6. Let Y be a random variable having a U(0, 1) distribution. We 
define the stochastic process {X (t), t > 0} by 

X(t) =e*t fort>0 


The first-order density function of the process can be obtained by using Propo- 
sition 1.2.2 (see Example 1.2.6): 


JEt) = fxe) = fry) [EEA 


-1-4 =+ if x € (t,te) 
fl z 


Next, the mean E|X(t)] of the process at time t > 0 is given by 
1 
E|X(t)] = / eMt-1ldy=t(e—1) fort >0 
0 
or, equivalently, by 


te 
EXO = f ode =te-t=1(e-1) for t > 0 
t 
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Finally, we have 
X(t)X(t+s) =e’t-e” (+s) =e”. t(t+3) 


It follows that 
e? — 





Rx(t,t+s) = E[X(t)X(t+ s)| = Ele?” t(t + s)| = t(t + 8) V s,t20 


2.2 Stationarity 


Definition 2.2.1. We say that the stochastic process {X(t),t € T} is sta- 
tionary, or strict-sense stationary (SSS), if its distribution function of 
order n is invariant under any change of origin: 


FE (Diewes Pe eas i ere stn) SF (Pics: aay Ul eS ETE „tn + 8) (2.8) 
for all s, n, and ty,... tn. 


Remark. The value of s in the preceding definition must be chosen so that 
tk +8 ET, fork =1,...,n. So, if T = |0,00), for instance, then t + s must 
be nonnegative for all k. 

In practice, it is difficult to show that a given stochastic process is station- 
ary in the strict sense (except in the case of Gaussian processes, as will be 
seen in Section 2.4). Consequently, we often satisfy ourselves with a weaker 
version of the notion of stationarity, by considering only the cases where n = 
1 and n = 2 in Definition 2.2.1. 

If {X(t),t € T} is a (continuous) SSS process, then we may write that 


f(a;t) = f(ajt+s) Ys,t (2.9) 
and 
f(®1,%93t1, to) = f(£1, £2;t1 + 8,t2 +8) V s,t1, te (2.10) 


We deduce from Eq. (2.9) that the first-order density function of the process 
must actually be independent of t: 


f(a;t) = f(x) Vt (2.11) 


Moreover, Eq. (2.10) implies that it is not necessary to know explicitly the 
values of tı and t2 to be able to evaluate f(x, %2;t1,t2). It is sufficient to 
know the difference tz — tı: 


f (x1, 225 t1,t2) = f (£1, £2;t2 — t1) V ti, te (2.12) 


In terms of the moments of the process, Eqs. (2.11) and (2.12) imply that 
mx(t) is a constant and that the autocorrelation function Rx (¢1,t2) is, in 
fact, a function Rẹ of a single variable: Rx (ti,t2) = Ry (te — tı). By abuse 
of notation, we simply write that Rx (ti,t2) = Rx (te — t1). 
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Definition 2.2.2. We say that the stochastic process {X(t),t € T} is wide- 
sense stationary (WSS) if mx(t) =m and 


Rx(ti,te) =Rx(te-ti1) Vtt ET (2.13) 


Remarks. i) Since mx(t) = m if {X(t),t € T} is wide-sense stationary, we 
can also write that 


Cx (t1, te) = Rx (ti, te) — mx(ti)mx (t2) = Rx (te — tı) — m = Cx (te i tı) 
2.14) 


V ti, t2 € T. Similarly, we have 


Cx (t2 —t 

px(t1,t2) = px(t2 — t1) = eH (2.15) 
ii) By choosing tı = t2 = t, we obtain that E[X?(t)] = Rx(t,t) = Rx (0), for 
all t € T. Therefore, the average power of a WSS s.p. does not depend on t. 
iii) We often take tı = t and tg = t+ when we calculate the function Rx (or 
Cx). If the process considered is WSS, then the function obtained depends 
only on s. 
iv) It is clear that an SSS stochastic process is also WSS. We will see in Section 
2.4 that, in the case of Gaussian processes, the converse is true as well. 


Example 2.2.1. The most important continuous-time and continuous-state 
stochastic process is the Wiener process, {W(t),t > 0}, which will be the 
subject (in part) of Chapter 4. We will see that E[W(t)] = 0 and that 


Cw(t,t +s) = Rw(t,t+s)=o't 


where ø > 0 is a constant and s,t > 0. Since the function Rw(t,t+s) depends 
on t (rather than on s), the Wiener process is not wide-sense stationary. 


Example 2.2.2. The Poisson process, that we denote by {N(t),t > 0} and that 
will be studied in detail in Chapter 5, possesses the following characteristics: 


E[N (t) = àt and Ry (ti, t2) = Amin{ty, te} 


for all t, tı, and tọ > 0, where À is a positive constant. It is therefore not 
stationary (not even in the wide sense), because its mean depends on t. If we 
define the stochastic process {X(t),t > 0} by 


Nit 
xa = 2 for t > 0 
then the mean of the process is a constant. However, we find that we can- 


not write that Rx(t1,te) = Rx(te — tı). Consequently, this process is not 
stationary either. 
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Remark. By definition, the Wiener and Poisson processes have stationary in- 
crements. However, as we have just seen, they are not even wide-sense sta- 
tionary. Therefore, these two notions must not be confused. 


Example 2.2.3. An elementary example of a strict-sense stationary stochastic 
process is obtained by setting 


X(t)=Y fort >0 


where Y is an arbitrary random variable. Since X(t) does not depend on the 
variable t, the process {X(t),t > 0} necessarily satisfies Eq. (2.8) in Definition 
22L 


Definition 2.2.3. The spectral density of a wide-sense stationary sto- 
chastic process, {X(t),t € T}, is the Fourier transform Sx(w) of its auto- 
correlation function: 


Sx(w) = f e I¥§ Ry(s)ds (2.16) 


— O&O 


Remarks. i) Inverting the Fourier transform, we obtain that 


OO 


Rx(s) = =j es Sy (w) dw (2.17) 


— OO 


ii) Since the autocorrelation function of a WSS process is an even function 
(that is, Rx(—s) = Rx(s)), the spectral density Sx(w) is a real and even 
function. We can then write that 


= E a 
Sx (w) = 2 | Rx(s)coswsds and Rx(s)= F Sx (w) cosws dw 
0 0 
(2.18) 
iii) It can be shown (the Wiener—Khintchin® theorem) that the spectral den- 


sity Sx (w) is a nonnegative function. Actually, a function Sx (w) is a spectral 
density if and only if it is nonnegative. 


Suppose now that the following relation holds between the processes 
{X(t),t€T} and {Y(t),t € T}: 


Y(t) = X(t) * h(t) = f E mere (2.19) 


3 Aleksandr Yakovlevich Khintchin, 1894-1959, was born and died in Russia. He 
contributed in a very important way to the development of the theory of stochas- 
tic processes. He was also interested in statistical mechanics and in information 
theory. 
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Remark. We can interpret the process {Y(t),t € T} as being the output of 
a linear system whose input is the process {X(t),t € T}. We write Y(t) = 
LIX). 

We assume again that {X(t),t € T} is stationary in the wide sense. If 
E|X(t)| = 0, we can show that {Y (t),t € T} is a WSS process with zero 
mean and such that 


Sy (w) = Sx(w)|H(w)|? (2.20) 
where 
Alo) c= f e7 h(s) ds (2.21) 
We also have 
2 E as 2 
EYO = Ry(0)=5- | Sx(w)|H(w)Pdw (20 (222 


2.3 Ergodicity 


In statistics, to estimate an unknown parameter of a distribution function, for 
example, the parameter A of an r.v. X having a Poi(A) distribution, we draw 
a random sample of X. That is, we take n observations, X1,...,Xn, of X 
and we assume that the X;’s have the same distribution function as X and 
are independent. Next, we write that the estimator \ of À (which is the mean 
of the distribution) is the arithmetic mean of the observations. Similarly, to 
estimate the mean mx (t) of a stochastic process {X(t),t € T} at time t, we 
must first take observations X (t, sk) of the process. Next, we define 


mx(t) = — > X(t, sx) (2.23) 
k=1 


Thus, we estimate the mean mx (t) of the s.p. by the mean of a random sample 
taken at time t. Of course, the more observations of the process at time t we 
have, the more precise the estimator m y(t} should be. Suppose, however, that 
we only have a single observation, X (t, s1), of X(t). Since we cannot estimate 
mx(t) in a reasonable way from a single observation, we would like to use 
the values of the process for the other values of t to estimate mx(t). For this 
to be possible, it is necessary (but not sufficient) that the mean mx(t) be 
independent of t. 


Definition 2.3.1. The temporal mean of the s.p. {X(t),t € R} is defined 
by 


S 
ia 5 J _X(ts)al (2.24) 
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Remarks. i) In this section, we will suppose that the set T is the entire 
real line. If T = (0,00), for example, we can modify the definition above. 
Moreover, in the discrete case, the integral is replaced by a sum. Thus, when 
T = {0,+1,+2,...}, we can write that 





ig N Xn. (8) (2.25) 
k 


where N is a natural number. 


ii) We call a realization or trajectory of the process {X(t),t € T} the graph 
of X(t, s) as a function of t, for a fixed value of s. 


Definition 2.3.2. The stochastic process {X(t),t € T} is said to be ergodic 
if any characteristic of the process can be obtained, with probability 1, from a 
single realization X(t,s) of the process. 


A stochastic process can be, in particular, mean ergodic, distribution er- 
godic (that is, ergodic with respect to the distribution function), correlation 
ergodic (with respect to the correlation function), etc. In this book, we will 
limit ourselves to the most important case, namely, the one where the process 
{X(t),t € T} is mean ergodic (see also p. 72). 


Definition 2.3.3. An s.p. {X(t),t € T} for which mx(t) =mVteET is 
called mean ergodic if 


P | Jim (X(t))5 = m| = (2.26) 


Now, the temporal mean, (X (t)}s, is a random variable. Since 


E|(X(t))s] = -5f E|X(t, s)| dt = — af. mdt = (2.27) 


we can state that Eq. (2.26) in the definition above will be satisfied if the 
variance V{(X(t))s] of the temporal mean decreases to 0 when S tends to 
infinity. Indeed, if lims_..., V[(X(t)) 5] = 0, then (X (t)}s converges to its mean 
value when S tends to infinity. To calculate the variance V[(X(t))s], we can 
use the following proposition. 


Proposition 2.3.1. The variance of the temporal mean of the stochastic pro- 
cess {X(t),t € T} is given by 


VUxtO)sl = gor | | J Cxltasta) dade (2.28) 
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Corollary 2.3.1. If the process {X(t),t € T} is wide-sense stationary, then 
we may write that 

1 28 


~ BS Jag XO) 


VX Es] 





s| 


Often, it is not necessary to calculate V[(X(t)}s] to determine whether 
the stochastic process {X(t),t € T} is mean ergodic. For example, when the 
process is WSS, we can use either of the sufficient conditions given in the 
following proposition. 


Proposition 2.3.2. The WSS s.p. {X(t),t € T} is mean ergodic if 


Cx(0)< oo and lim Cx(s)=0 (2.30) 


|s| 00 


or if its autocovariance function C'x(s) is absolutely integrable, that ts, af 


D [Cx(s)| ds < œ (2.31) 


Example 2.3.1. The elementary stochastic process defined in Example 2.2.3 is 
strict-sense stationary. However, it is not mean ergodic. Indeed, we have 


S 
(X(t))5 = ag | Y#=Y 


Thus, we may write that V|(X(t)) 5] = V[Y]. Now, if Y is not a constant, the 
variance V[Y] is strictly positive and does not decrease to 0 when S tends to 
infinity (since V/Y} does not depend on S). Therefore, an arbitrary stochastic 
process can be mean ergodic without even being WSS (provided that mx (t) 
is a constant), and a strict-sense stationary process is not necessarily mean 
ergodic. 


Example 2.3.2. As will be seen in Chapter 5, the s.p. {X(t),¢t € T} called the 
random telegraph signal, which is defined from a Poisson process, is zero mean 
and its autocovariance function is given by 


Ox (s) =f e7 Als! 


where A is a positive constant. Using the conditions in Eq. (2.30), we can state 
that the process is mean ergodic. Indeed, we have 


Cx(0)=1<0o and lim Cy(s)= lim e793! =0 


[| s|—+00 |s]—00 


Actually, we also have 
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J ICx(s)|ds=2 | dai R < OO 
0 


— 00 


Remarks. i) It is important to remember that the conditions in Proposition 
2.3.2 are sufficient, but not necessary, conditions. Consequently, if we cannot 
show that the process considered is mean ergodic by making use of this propo- 
sition, then we must calculate the variance of the temporal mean and check 
whether it decreases to 0 or not with S —> oo. 


ii) It can be shown that the random telegraph signal is an example of a strict- 
sense stationary stochastic process. Therefore, we can calculate the variance 
V|(X(t))s] by using the formula (2.29): 


VI(X(t))5] = 5 | P> RAs a 


The integral above is not difficult to evaluate. However, here, it is not even 
necessary to calculate it explicitly. It is sufficient to replace the expression 
between the square brackets by 1, because this expression is comprised of 
values between 0 and 1 when s varies from 0 to 25. It follows that 


1— e748 
VUX(E)s] < aye 
Finally, we have 
T= eT 4AS 
geo ANRE ag 


Since the variance V[|(X(t)}s] is nonnegative, we can conclude that 


which confirms the fact that the random telegraph signal is mean ergodic. 


2.4 Gaussian and Markovian processes 


The bivariate normal distribution was defined in Example 1.3.6. The general- 
ization of this distribution to the n-dimensional case is named the multinormal 
distribution. 


Definition 2.4.1. We say that the random vector (X1,... , Xn) has a multi- 
normal distribution if each random variable X, can be expressed as a linear 
combination of independent random variables Z,,... , Zm, where Z; ~ N(0,1), 
forj =1,...,m. That is, if 


m 
Xk=uk+Y ckjZj fork=1,...,n (2.32) 
j=1 


where pi, is a real constant, for all k. 
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Just as a Gaussian distribution N(u, g?) is completely determined by its 
mean u and its variance g?, and a bivariate normal distribution by its pa- 
rameters Hx, HY, ae. ot and p, the joint density function of the random 
vector X = (Xj,...,X,) is completely determined by the vector of means 
m := (4x,;--. ,{x,,) and the covariance matrix K, where 


VX] Cov[X1, Xo] ... Cov[X1, Xn] 


Cov[ Xo, X1] V[Xq]  ... Cov[ X2, Xn] 


K: (2.33) 


Cov| Xn, Xi! Cov| Xn, Xe]... VEX] 


By analogy with the one-dimensional case, we write that X ~ N(m, K). 


The matrix K is symmetrical, because Cov| X,Y] = Cov[Y, X], and non- 
negative definite: 


n n 
NOY cickCov[X;, X] >20 V cick ER (2.34) 
t=1 k=1 


If, in addition, it is nonsingular, then we may write that 


= a : eee TG =n 


for x := (1,...,%,) € R”, where “det” denotes the determinant and 7 
denotes the transpose of the vector. 


Proposition 2.4.1. Let X ~ N(m,K). The joint characteristic function of 
the r.v. X: 


x(w... ,Wn) := E [exp{j (wi Xi +... + wnn) (2.36) 
is given by 
i ! n 1 n m l 1 
bx (w) = exp l; D -3 du ous = exp {mut _ zokor) 
(2.37) 
where Cik := Cov|X;, Xx] and w := (wy,... Wn). 


Proof. We use the fact that any linear combination of Gaussian random 
variables also has a Gaussian distribution. More precisely, we can write that 


Y := w Xi +... +wnXn ~ N(py, 0%) (2.38) 


where 


60 2 Stochastic Processes 


n n n 
pyas Ņ wittx, and of := NO Y wwe oir (2.39) 


i=l i=1 k=1 


We obtain the formula (2.37) by observing (see Table 1.1, p. 19) that 


ọx(w) = Ele”) = ġy (1) = exp (jux = 50?) z (2.40) 


Properties. i) If Cov[X;, Xg] = 0, then the random variables X; and X, are 
independent. 

ii) If Y;, for i =1,...,m, is a linear combination of the random variables X;, 
of a random vector (Xj,..., Xn) having a multinormal distribution, then the 
Pv (CY leas , Ym) also has a multinormal distribution. 


Example 2.4.1. Let X = (X1,... , Xn) be a random vector having a multinor- 
mal distribution N(0,I,,), where 0 := (0,... ,0) and I, is the identity matriz 
of order n. Thus, all the random variables X% have a standard Gaussian dis- 
tribution and are independent (because oj; = 0 V i # 7). It follows that the 
mathematical expectation of the square of the distance of the vector X from 
the origin is 


n n 
EIX? + X3+...4X2) => aA SIn 
k=1 k=1 
and the variance of the squared distance is 
VIX? + XP +.. +X So vpxe'S So 2=2n 
k=1 k=1 
because if Z ~ N(0,1), then we have 
4 4 d /2 
Bea] | aş 
a) E 


so that V[Z?] =3 — 1? =2. 


Definition 2.4.2. A stochastic process {X(t),t € T} is said to be a Gaus- 
sian process if the random vector (X(t1),...,X(tn)) has a multinormal dis- 
tribution, for any n and for all ti,... ,tn- 


Remark. Let X be a random variable whose distribution is N(ux,0%). Any 
affine transformation of X also has a Gaussian distribution: 


Y :=aX +b => Y ~Nl(aux +b,070%) (2.41) 
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Similarly, any affine transformation of a Gaussian process remains a Gaussian 
process. For example, if {X(t),t € T} is a Gaussian process, then the s.p. 
{Y(t),t € T} defined by 


Y(t) =2X(t)-1 or Y(t) =X(t*) (2.42) 
is Gaussian as well. We can also show that {Y (t),t € T} is a Gaussian process 
if 

t 
Ye) f X (s) ds (2.43) 
0 


because an integral is the limit of a sum. However, the process is not Gaussian 
if 
Y(t)= X(t) or Y(t) =e*” (2.44) 


etc. 


Proposition 2.4.2. If a Gaussian process {X(t),t € T} is such that its mean 
mx (t) is a constant mx and if its autocovariance function Cx (t,t+s) depends 
only on s, then it is stationary (in the strict sense). 


Proof. Since the nth-order characteristic function of the s.p. {X(t),¢ € T} is 
given by [see Eq. (2.37)] 


n n n n 
: , I 
E = f Yaxta) = exp [jms dei a D ` Cx (ti — aua} 
t=1 i=1 t=1 k=1 
(2.45) 
we can assert that the statistical characteristics of a Gaussian process de- 


pend only on its mean and its autocovariance function. Now, we see that the 
function above is invariant under any change of time origin. O 


Remark. The preceding proposition means that a wide-sense stationary Gaus- 
sian process is also strict-sense stationary. 


Definition 2.4.3. An s.p. {X(t),t € T} is said to be Markovian if 
P[X (tn) < am | X(t),t < tn-1] = PIX (tn) < n | X(tn-1)] (2-46) 
where tni < tn. 


Remarks. i) We say that the future, given the present and the past, depends 
only on the present. 


ii) If {X(t),t € T} is a discrete-state process, we can write the formula (2.46) 
as follows: 
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PlXe+0 = j | Xi = i, Xin = tn... My = i) = PiXie = j | Xe =i] (2.47) 
for all states 71,... ‚in, i, Jj, and for all time instants tı < ... < tn < t and 


s >Q. 


iii) The random walk considered in Example 2.1.1 is a typical example of a 
Markovian process, which follows directly from the fact that we assume that 
the tosses of the coin are independent. 


A Markovian, continuous-time, and continuous-state stochastic process, 
{X(t),t € T}, is completely determined by its first-order density function: 


flit) = —PIX(t) <a] (2.48) 
and by its conditional transition density function, defined by 
p(x, £o; t, to) = Fx (t)|X (to) (© | xo) (2.49) 
_ lim P|X(t) € (2,2 +e | X(to) = xo] nee 


Since the process must be somewhere at time t, we have 


oO oO 
f f(a;t)dx=1 and / p(x, x0;t, to) dx = 1 (2.50) 
as -%0 


Moreover, by conditioning on all possible initial states, we may write that 
OO 
flat) = f f (xo; to) p(x, £o; t, to) dzo (2.51) 
— 00 


Similarly, we deduce from the Chapmant-Kolmogorov equations (see 
Chapter 3) that 


OO 


p(x, x93 t, to) = | plz, 21; t, t1)p(r1, 203 t1, to) dry (2.52) 


where to < tı < t. Finally, since at the initial time the distribution of the 
process is deterministic, we also have 


4 Sydney Chapman, 1888-1970, was born in England and died in the United States. 
He is especially known for his work in geophysics. One of the craters of the moon 
is named after him. 

5 Andrei Nikolaevich Kolmogorov, 1903-1987, was born and died in Russia. He was 
a great mathematician who, before getting his Ph.D., had already published 18 
scientific papers, many of which were written during his undergraduate studies. 
His work on Markov processes in continuous time and with continuous-state space 
is the basis of the theory of diffusion processes. His book on theoretical probability, 
published in 1933, marks the beginning of modern probability theory. He also 
contributed in an important way to many other domains of mathematics, notably 
to the theory of dynamical systems. 
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lim p(z, To; t, to) = d(x — Zo) (2.53) 
Q 


where 6(-) is the Dirac? delta function defined by 


_ jf 0 ifz 0 
ô(x) = a ee (2.54) 
so that 
OO 
/ d(x) dx =1 (2.55) 
—oo 


Definition 2.4.4. The infinitesimal mean m(zx;t) and the infinitesimal 
variance u(z;t) of the continuous-time and continuous-state stochastic pro- 
cess {X(t),t € T} are defined, respectively, by 
1 
mind) = lim —-E[X(t+6)—- X(t) | X(t) = zq] (2.56) 
€ € 
and 


v(z;t) = lim -E(XG +6) -X(t))? | X(t) =2] (2.57) 


Remarks. i) We can also obtain m(zo; to) and v(xo; to) as follows: 


7 ð — £ 
m(xo;to) = lim z EJ X(t) | X (to) = Zo] (2.58) 
and 
v(toto) = l 2 OR = (2.59) 
PEO ai ei l 


ii) Suppose that the process {X(t),t € T} has infinitesimal moments m(z;t) 
= m(x) V t and v(x; t) = v(x) and that its state space is the interval [a,b] (or 
la, b), etc.). Let 


Y(t) := g[X(t)] forte T (2.60) 


If the function g is strictly increasing or decreasing on the interval [a,b] and 
if the second derivative g”(x) exists and is continuous, for a < x < b, then 


8 Paul Adrien Maurice Dirac, 1902-1984, was born in England and died in the 
United States. He was a physicist whose father was a French-speaking Swiss. He 
won the Nobel Prize for physics in 1933 for his work on quantum mechanics. He 
was a professor at the University of Cambridge for 37 years. 
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we can show that the infinitesimal moments of the process {Y (t),t € T} are 
given by 


my (y) = m(a)g'(x) + Zvle)g"(2) (2.61) 
and 


vy (y) = o(z)[9'(2))° (2.62) 


where x = g~(y). Moreover, the state space of the process is the interval 
(g(a), g(b)] (respectively, [g(b), g(a)]) if g is strictly increasing (resp., decreas- 


kd 


ing). 


It can be shown that the function p(x, £o; t, to) satisfies the following par- 
tial differential equations: 


Op ð 1 8? | 
at De [m(x; t) p] > pz lv(x; t) p] (2.63) 
and 
Op Op 1 07 
ry a + — 2.64 
ato 5 HOO) A T z (Z0; to) Au? 0 ( ) 


These equations are called the Kolmogorov equations or the diffusion equa- 
tions. The first one is the Kolmogorov forward equation (or Fokker’ —Planck® 
equation), and the second one is the Kolmogorov backward equation. 


Definition 2.4.5. If the function p(x, x9;t,to) depends only on (x, zo and) 
the difference t — tg, the stochastic process {X(t),t € T} is said to be time- 
homogeneous. 


Remarks. i) If the s.p. {X(t),t € T} is time-homogeneous, then the functions 
m(x;t) and v(z;t) do not depend on t. 

ii) Note that a time-homogeneous s.p. is not necessarily (wide-sense) station- 
ary, because the function f(z;t) may depend on the variable t. On the other 
hand, if f(x;t) = f(x), then the process is strict-sense stationary, because it 
is completely determined by f(z;t) and p(x, £o; t, to). 


Definition 2.4.6. A stochastic process {B(t),t > 0} is called a white noise 
(process) if its mean is equal to zero and if its autocovariance function is of 
the form 


Cpg(ti, t2) = q(tı) b(t2 — tı) (2.65) 
where q(t1) > 0 and 6(-) is the Dirac delta function (see p. 63). 


7 Adriaan Daniël Fokker, 1887-1972, was born in Indonesia and died in the Nether- 
lands. He was a physicist and musician. He proved the equation in question in 
1913 in his Ph.D. thesis. 

8 Max Karl Ernst Ludwig Planck, 1858-1947, born and died in Germany, was a 
renowned physicist famous for the development of quantum theory. 
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Remark. Actually, {B(t),t > 0} is not a stochastic process in the proper sense 
of the term. 


Finally, the following important result can be shown. 


Proposition 2.4.3. A Gaussian and stationary process, {X(t),t € T}, is 
Markovian if and only if its autocorrelation function is of the form 


Rx(s) = 07e7 5! (2.66) 
where o and g are positive constants. 


Example 2.4.2. The Wiener process {W(t),t > 0} (see Example 2.2.1) is such 
that EJW (t) | W(to) = wo] = wo and V[W (t) | W(to) = wo} = o7(t — to). 
It follows that its infinitesimal mean and variance are given by m(w;t) = 
0 and v(w;t) = o*. Therefore, the conditional transition density function 
p(w, wWo;t, to) satisfies the Kolmogorov forward equation 


Op. 183 

ot 5 ow? P= 
We can check that W(t) | {W (to) = wo} has a Gaussian distribution with 
parameters wo and o7(t — to). That is, the function p(w, wo;t, to) given by 


1 1 (w — wo)? ) 
p(w, wo; t, to) = —=——== ex r for t > to 
l ) 27a2(t — to) Ag a*(t — to) 


is a solution of the partial differential equation above. Moreover, we have 


lim p(w, wo; t, to) = d(w — wo) 
tlto 


as required. 


2.5 Exercises 


Section 2.1 


Question no. 1 
We define the stochastic process {X (t), t > 0} by 


X(t)=e"** fort>0 
where Y is a random variable having a uniform distribution on the interval 
(0,1). Calculate 
(a) the first-order density function of the process {X (t), t > 0}, 
(b) E [X(t)], for t > 0, 
(c) Cx (t,t + s), where s,t > 0. 
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Question no. 2 : 

Let {Xn,n = 1,2,...} be a Bernoulli process. That is, the random vari- 
ables X1, X2,... are independent and all have a Bernoulli distribution with 
parameter p. Calculate 


(a) the particular case p(0,1;n, = 0,n2 = 1) of the second-order probability 
mass function of the process, 


(b) the correlation coefficient px (n,m) of the process if p = 1/2 and n,m € 
io ee 


Question no. 3 
Calculate the first-order density function of the s.p. {X(t),¢ > 0} defined 
by 
X(t) =tY +1 


where Y is a random variable having a U(0,1) distribution. 


Question no. 4 
We define the stochastic process {X(t),t > 0} by 


X(t) = for t>0 
where Y has a uniform distribution on the interval (0,2). Calculate the func- 
tion f(x;t), for x > t/2. 


Question no. 5 

We consider the process {X(t),t > 0} defined by X(t) = (tan Y)t, for 
t > 0, where Y is a random variable having a uniform distribution on the 
interval (—7/2, 7/2). Calculate the probability P{[3 t € (0,1): X(t) ¢ [0,11]. 
In other words, calculate the probability that the process {X (t), t > 0} leaves 
the interval [0,1] between 0 and 1. 


Indication. It can be shown that the r.v. W := tan Y has the following density 
function: 
= forweR 
fu(w) =a for w 
That is, W has a Cauchy? distribution, or a Student'® distribution with one 
degree of freedom. 


°’ Augustin Louis Cauchy, 1789-1857, was born and died in France. He is considered 
the father of mathematical analysis and the inventor of the theory of functions 
of a complex variable. 

10 Pseudonym of William Sealy Gosset, 1876-1937, who was born and died in En- 
gland. He worked as a chemist for the Guinness brewery, in Ireland, where he 
invented a statistical test for the control of the quality of beer. This test uses the 
distribution that bears his name. 
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Question no. 6 
Are the increments of the stochastic process {X(t),t > 0} defined in Ex- 
ample 2.1.6 independent? stationary? Justify. 


Question no. 7 

(a) Find the autocorrelation function of the process {X (t), t > 0} defined by 
X(t)=1ifY >tand X(t) =Oif Y <t, where Y is a random variable having 
a U(0,c) distribution. 

(b) Calculate Rx (tı, t2) in (a) if X(t) = 6(Y — t) instead, where 6(-) is the 
Dirac delta function. 


Question no. 8 

We consider the process {X(t),t > 0} defined by X(t) = e~**, for t > 0, 
where Y is a continuous random variable whose density function is fy (y), for 
y = 0. 


(a) Find f(a;t) in terms of fy (y). 
(b) Calculate E|X(t)] and Rx(t1,t2) when Y has an exponential distribution 
with parameter 1. 


Question no. 9 
Let 


ra= || Bar ds fort>0 


where {B(t),t > 0} is the white noise process defined on p. 64. What is the 
average power of the stochastic process {X(t),t > 0}? 


Section 2.2 


Question no. 10 
Is the stochastic process {X (t), t > 0} defined in Question no. 1 wide-sense 
stationary? Justify. 


Question no. 11 
Let {X(t),t > 0} be a stochastic process whose autocorrelation and auto- 
covariance functions are 


Rx (t1,t2) = ela-t2l4i4 and Cx (ti, t2) = elt —te| 


Is the process wide-sense stationary? Justify. 


Question no. 12 
Let {X(t),t > 0} be a wide-sense stationary stochastic process, with zero 
mean and autocorrelation function given by Rx(s) = e7!*!. We define 


Y(t) =tX7(1/t) fort >0 


Is the stochastic process {Y (t),t > 0} wide-sense stationary? Justify. 
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Question no. 13 
We consider a stochastic process {X (t), t > 0} for which 


1 


Cx (t1, te) = toaa 


1 
TEREN and Ryx(ti, t2) = 


Is the process wide-sense stationary? Justify. 


Question no. 14 
Let Y be a random variable having a uniform distribution on the interval 
(—1,1). We consider the stochastic process {X (t), t > 0} defined by 


X(t)=Y°t fort >0 


Is the process stationary? J ustify. 


Question no. 15 
Let Y be a random variable such that ġy (1) = 2¢y(—1) and ¢y(2) = 
Ady (—2), where gy(-) is the characteristic function of Y. We set 


X(t) =cos(wt+Y) fort >0 


Show that the stochastic process {X(t),t > 0} is wide-sense stationary. 


Indication. We have the following trigonometric identity: 


1 
cos(wt; + s) cos(wtz + s) = = {cosw(t; — te) + cos(wt, + wtz + 2s)} 
2 


Question no. 16 
We set 
Y(t)=X(t+1)—X(t) forteR 


where {X(t),t € R} is a wide-sense stationary stochastic process. Find the 
spectral density of the process {Y (t),t € R} in terms of Sx (w). 


Question no. 17 
Consider the wide-sense stationary process {X(t),t € R} whose spectral 
density is Sy(w) = 2/(1+ w?). We set 
t+1 
Yo) = X(s)ds forte R 


Calculate Sy (w). 
Indication. Write Y (t) in the form 


Y(t) = f h(t — s)X (s) ds 


for an appropriately chosen function h. 
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Question no. 18 
We consider the random variable Y (t) defined by Eq. (2.19). Calculate the 
mean of X(t)Y (t) if {X(t),t € T} is a white noise process. 
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Question no. 19 
Are the stochastic processes defined in Questions nos. 11, 12, and 13 mean 
ergodic? Justify. 


Question no. 20 
Is the Bernoulli process defined in Example 2.1.5 mean ergodic? Justify. 


Question no. 21 
We define X(t) = m+ B(t), for t > 0, where m is a constant and { B(t),t > 
0} is a white noise process, so that E[B(t)] = 0 and 


Cp(t1,t2) = q(t) d(te — tı) 


where 6(-) is the Dirac delta function. Show that the process {X(t),t > O} is 
mean ergodic if the function q is bounded. 


Question no. 22 
Let {X(t),t > 0} be a wide-sense stationary stochastic process, with zero 
mean and for which 
Rx(s) = e~ 5! cos 2rws 


where a and w are positive constants. Is this process mean ergodic? Justify. 


Question no. 23 
The stochastic process {X (t), t > 0} is defined by 


X(t) =Y + Bit) 


where Y ~ N(0,1) and {B(t),t > 0} is a white noise process whose autocor- 
relation function is Rg(s) = cd(s), with c > 0. We assume that the random 
variable Y and the white noise are independent. Is the process {X(t),t > 0} 
(a) wide-sense stationary? 

(b) mean ergodic? 


Question no. 24 
Let {B(t),t > 0} be a white noise process for which the function q(t) is 
a constant c > 0. Calculate the variance of the random variable 
1 pT 


Ir = oT Pa dt 
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Question no. 25 

The standard Brownian motion is the particular case of the Wiener process 
{W(t),t > 0} (see Example 2.2.1) for which E[W (t)] = 0 and Cw(t1,ta) = 
min{ty, to}. 
(a) Calculate V|[(W(t)) 5]. 


(b) Is the process mean ergodic? Justify. 


Section 2.4 


Question no. 26 

Let (X1,... , Xn) be a random vector having a multinormal distribution 
for which m = (0,... ,0) and the covariance matrix K is the identity matrix 
of order n (> 4). 
(a) Calculate the characteristic function of Y := X1 + X2. 
Reminder. The characteristic function of a random variable X having a Gaus- 
sian N(y,07) distribution is given by ¢x(w) = exp (juw — $07w?). 


(b) Does the random vector (Y, Z), where Z := X3 — X4, have a bivariate 
normal distribution? If it does, give its five parameters; otherwise, justify. 


Question no. 27 
Let {X(t),t > 0} be a Gaussian process such that E|.X(t)| = ut, for t > 0, 
where u is a nonzero constant, and 


Rx(t,t+s) =2t+ p*t(t+s) for s,t>0 
We define Y(t) = X(t) — pt, for t > 0. Is the process {Y (t),t > 0} stationary? 
Justify. 


Question no. 28 
We consider a Gaussian process {X(t),t > 0} for which E[X(t)] = 0 and 
whose autocovariance function is 


Cx(t,t+s)=e" for s,t>0 
Let Y(t) := X?(t), for t > 0. Is the stochastic process {Y(t),t > 0} (a) sta- 
tionary? (b) mean ergodic? Justify. 


Question no. 29 

Let {Xn,n = 1,2,...} be a discrete-time and discrete-state process such 
that 
P Xarea Anr Age = (ase 


a P|Xn+1 =J | Xn = 1, Xn-1 = a 


for all states i1,...,%,-1,i,j and for any time instant n. This process is not 
Markovian. Transform the state space so that the process thus obtained is 
Markovian. 
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Question no. 30 
Suppose that {X(t),t > 0} is a Gaussian process such that 


EIX (t) | X(to) = xo] = Xo and V[X(t) | X(to) = zo] = t Vt>0 


Let 
Y(t):=e*™ fort >0 


(a) Calculate the infinitesimal parameters of the process {Y(t),¢ > 0} by 
using the formulas (2.56) and (2.57). 
(b) Check your results with the help of the formulas (2.58) and (2.59). 


Question no. 31 
Find a solution of the Fokker—Planck equation 


ð ð 1 8? 
n i 5, HP) me er ae —0 _ (for x €R and t > to > 0) 


where u € R and o > 0 are constants, for which 


lim p(x, £o; t, to) = d(x — xo) 
tito 


Indications. i) Try the density function of a Gaussian distribution as a solution. 


ii) We can take the Fourier transform (with respect to x) of the equation and 
then invert the solution of the ordinary differential equation obtained. We 
have 


lim p(z, £o;t, to) =0 
x—too 


which implies that 
O 
A, Pls: To; t, to) =0 


im 
~L— +00 
as well. 
Question no. 32 
Let {X(t),t € R} be a wide-sense stationary Gaussian process, with zero 
mean and Rx(s) = 2e~4l*!, We define Y = X(t + s) and Z = X(t — s), where 
s> 0. 


(a) What is the distribution of the random variable X(t)? 
(b) Calculate the mean of YZ. 
(c) Calculate the variance of Y + Z. 


Question no. 33 
The Gaussian process {X(t),t > 0} is such that E[X(t)] = 0 Y ¢ and 


Rx (t1,t2) = cpe 2-2 


where c1, C2 > 0. 
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(a) Show that the stochastic process {X(t),t > 0} is stationary and mean 
ergodic. 


(b) We define the process {Y (t), t > 0} by 


Lif X(t) < 
Y= 4 oe xu) oe 


We then have E[Y(¢)] = F(a;t). If the process {X(t),¢ > 0} is stationary, 
we may write that F(z;t) = F(x). We say that the s.p. {X(t),t > 0} is 
distribution ergodic if {Y (t),t > 0} is mean ergodic. Is the stochastic process 
{X (t), t > 0} distribution ergodic? 

(c) When the process {X(t),t > 0} is stationary, we also define the process 
{Z,(t),t > 0} by Z,(t) = X(t)X(t+s), where s > 0. It follows that E/Z,(t)| = 
Rx(s). We say that the s.p. {X (t), t > 0} is correlation ergodic if {Z,(t),t = 0} 
is mean ergodic. Can we assert that {X(t),t > 0} is correlation ergodic? 


Question no. 34 
Suppose that {X(t),t € R} is a stationary Gaussian process such that 


Cx(s) = Rx(s) =e?" forseR 


(a) What is the distribution of X(t)? Justify. 
(b) We define the random variable 


Y(t) = &[X(t)] +1 
where ®(-) is the distribution function of the N(0,1) distribution. That is, 
(z) := P[N(0,1) < z}. What is the distribution of Y (t)? Justify. | 
Indication. The inverse function Ø`! exists. 


Question no. 35 
Let {X(t),t € R} be a stationary Gaussian process whose autocovariance 


function is 11 — (|s|/2)] if |s| <2 
All — (is|/2) if |s| < 
Cx(s) = { 0 if |s| > 2 
Is the stochastic process {X(t),t € R} 
(a) mean ergodic? 
(b) distribution ergodic (see Question no. 33)? Justify. 


3 





Markov Chains 


3.1 Introduction 


The notion of a Markovian process was seen in Section 2.4. In the general 
case, the stochastic process {X(t),t € T} is said to be Markovian if 


PIX Gag) Al X(t) Sant e a = PX Ga) CAL XG) Hee]. 12.7) 


for all events A and for all time instants tp < ty41. 


Equation (3.1) means that the probability that the process moves from 
state Tt„, where it is at time tn, to a state included in A at time tn+1 does not 
depend on the way the process reached z+, from £, where to is the initial 
time (that is, does not depend on the path followed by the process from Tt 
to Lt, ). 


In this chapter, we will consider the cases where {X (t), t € T} is a discrete- 
time process and where it is a continuous-time and discrete-state process. 
Actually, we will only mention briefly, within the framework of an example, 
the case of discrete-time and continuous-state processes. 


When {X,,n = 0,1,...} is a discrete-time and discrete-state process, the 
Markov property implies that 


Pi Xna 9 Age = EAS pg Ao So) Se PiX SF | eT 


(3.2) 
for all states ig,... ,%,~1,7,j, and for any time n. We also have 
PX aS et Sey ne = ny gS o] 
= P[|Xn+1 =j | Xn-1 = in—1| (3.3) 


etc., which means that the transition probabilities depend only on the most 
recent information about the process that is available. 
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Remark. We call a Markov chain a discrete-time process that possesses 
the Markov property. Originally, this expression denoted discrete-time and 
discrete-state Markovian processes. However, we can accept the fact that 
the state space of the process may be continuous. By extension, we will call 
a continuous-time Markov chain a discrete-state (and continuous-time) 
Markovian stochastic process. Discrete-time and continuous-time Markov 
chains will be studied in Sections 3.2 and 3.3, respectively. 


We can generalize Eq. (3.2) as follows. 


Proposition 3.1.1. If the discrete-time and discrete-state stochastic process 
{Xn n =0,1,...} possesses the Markov property, then 


P(X ni Anpa) EB | Xn = tn, Xn-1 = tn-1,--- , Xo = io] 
= P|(Xn41, Xn42;- ; .) E€ B | An in] (3.4) 


where B is an infinite-dimensional event. 


Remark. For example, by using the fact that 
PIANBNC)  P[ANBNC] PIBNC] 
PIC] —  PIBNC P|C| 
= P[A| BNCIP[B| C] (3.5) 


P|ANB|C]= 


we can easily show that 


Pts Ango) = Gn iy ni2) | Ae = nA eS ge = i] 
= Pl Anaa) = (ia) | An in] (3.6) 


We now give several examples of discrete-time and continuous-time Markov 
chains. 


Example 3.1.1. One of the simplest, but not trivial, examples of a Markovian 
process is that of the random walk (see p. 48). We can generalize the random 
walk as follows: if the particle is in state i at time n, then it will be in state 
j at time n + 1 with probability 


pif j=i+1 
' gif j=i=i 
P| Xn = j | Xn = i] = rifj=i 


0 otherwise 


where p, q, and r are nonnegative constants such that p +q +r = 1. That 
is, we add the possibility that the particle does not move from the position it 
occupies at time n. 

Another way of generalizing the random walk is to assume that the prob- 
abilities p and q (and r in the case above) may depend on the state i where 
the particle is. Thus, we have 
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P| Xn =J| Xn == 4 Gifg=i-l (3.7) 
0 otherwise 


and p; + q = 1, for any i € {0, +1, 2,... }. 

Finally, we can generalize the random walk to the d-dimensional case, 
where d € N. For example, a two-dimensional (d = 2) random walk is a 
stochastic process {(Xn, Yn), n = 0,1,...} for which the state space is the set 


Scx,.y,) = (9): 4,9 € {0, £1, £2,...}} 


and such that 
Pi Xe Ynti) z (Gatis Ingi) | (Xas Yn) Pe asda) 


pı if ingt = in +1, jn+1 = Jn 
P2 if int = În, Jn+i = Jn +1 

= 4 qi İf ingt = în — 1, Jn+1 = Jn 
q2 if inti = În, In+1 T In ae 
0 otherwise 


where pı + p2 +91 + q2 = 1 (and p; > 0 and q; > 0, for i = 1,2). Thus, the 
particle can only move to one of its four nearest neighbors. In three dimensions, 
the particle can move from a given state to one of its six nearest neighbors [the 
neighbors of the origin being the triplets (£1,0,0), (0, +1,0) and (0,0, +1)}, 
etc. 


Example 3.1.2. An important particular case of the random walk having the 
transition probabilities given in Eq. (3.7) is the one where 
AN = i 
P= -N and qi= WN 
If we suppose that the state space is the finite set {0,1,... , N}, then we can 
give the following interpretation to this random walk: we consider two urns 
that contain N balls in all. At each time unit, a ball is taken at random from 
all the balls and is placed in the other urn. Let X, be the number of balls in 
urn I after n shifts. This model was used by Paul and Tatiana Ehrenfest! to 
study the transfer of heat between the molecules in a gas. 


Example 3.1.8. Suppose that we observe the number of customers that are 
standing in line in front of an automated teller machine. Let Xn be the number 
of customers in the queue at time n € {0,1,...}. That is, we observe the 
system at deterministic time instants separated by one time unit, for example, 


' Paul Ehrenfest, 1880-1933, born in Austria and died in the Netherlands, was 
a physicist who studied statistical mechanics and quantum theory. His wife, of 
Russian origin, was Tatiana Ehrenfest-Afanaseva, 1876-1964. She was interested 
in the same subjects. 
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every hour, or every 10 minutes (a time unit may be an arbitrary number of 
minutes), etc. Suppose next that if a customer is using the teller machine at 
time n, then the probability that he is finished before time n + 1 is q € (0,1). 
Finally, suppose that the probability that k customers arrive during a time 
unit is given by 
ant 
PY, =kl =e Ti | 

for k = 0,1,... and for any n € {0,1,...}. That is, the number Y,, of arrivals 
in the system in the interval [n,n +1) has a Poisson distribution with parame- 
ter \ Vn. Thus, we may write that Xn+1 = Xn + Yn with probability p = 1—g¢ 
and Xn41 = (Xn —1)+Yn with probability q. We find that {Xn, n = 0,1,...} 
is a (discrete-time and discrete-state) Markov chain for which the transition 
probabilities are 


aA” pO ASPE ouii g l 
Re OS e 
ZA a oe l 
FA d 1 a gare ifj=i-landieN 
Rr, if j =k andi=0 
0 otherwise 
(3.8) 


This is an example of a queueing model in discrete-time. 


Example 3.1.4. An example of a discrete-time, but continuous-state, Markov 
chain is obtained from a problem in optimal control: suppose that an object 
moves on the real line. Let X, be its position at time n € {0,1,...}. We 
suppose that 

Xn+1 = AnXn + OnUn + €En 


where a, and b, are nonzero constants, un is the control (or command) vari- 
able, and €n is the term that corresponds to the noise in the system. We 
assume that en ~ N(0,07), for all n and that the random variables X„ and 
En are independent. The objective is to bring the object close to the origin. 
To do so, we choose 

a 


Ke Vn (3.9) 


Un = 


Remark. In stochastic optimal control, the variable un must be chosen so as 
to minimize the mathematical expectation of a certain cost function J. If the 
function in question is of the form 


OO 


1 
I(XnsUn) = >> (ex; He u) 


n=0 


where c > 0 and k are constants, then the optimal solution is indeed obtained 
by using the formula (3.9) above. 
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The s.p. {Xn n =0,1,...} is a Markov chain. If fx„,,|x„(Y | 2) denotes 
the conditional density function of the random variable Xn+1, given that 
Xn = x, we may write that 


1 y? 
= —— — -> fory E R 
Ti a] oe { 552 y 
That is, Xy41 | Xn ~ N(0,07). 

We can also suppose that the noise €n has a variance depending on Xn. For 
instance, Ven] = ao? X2. In this case, we have that Xn+1 | Xn ~ N(0,0°X2). 


Example 3.1.5. Suppose that X(t) denotes the number of customers who ar- 
rived in the interval [0, t] in the example of a queueing system in discrete-time 
(see Example 3.1.3). If we assume that the time 7, elapsed until the arrival 
of a new customer into the system, when there have been n arrivals since 
the initial time, has an exponential distribution with parameter 4, for all n, 
and that the random variables To, 71,... are independent, then we find that 
{X (t), t > 0} is a continuous-time and discrete-state Markovian process called 
a Poisson process. In this case, the number of arrivals in the system in the 
interval [0, t] has a Poisson distribution with parameter At. Therefore, during 
one time unit, the number of arrivals indeed has a Poi(A) distribution. 

In general, if the random variable 7; has an exponential distribution with 
parameter Ay, then {X(t),t > 0} is a continuous-time Markov chain. 


Example 3.1.6. In the preceding example, we considered only the arrivals in 
the system. Suppose that the time an arbitrary customer spends using the 
automated teller machine is an exponential random variable with parameter 
u. Now, let X(t) be the number of customers in the system at time t > 0. 
If the customers arrive one at a time, then the process {X(t),t > 0} is an 
example of a continuous-time Markov chain called a birth and death process. It 
is also the basic model in the theory of queues, as will be seen in Chapter 6. It 
is denoted by the symbol M/M/1. When there are two servers (for example, 
two automated teller machines) rather than a single one, we write M/M/2, 
etc. 


Remark. Actually, the Poisson process is a particular case of the so-called pure 
birth (or growth) processes. 


3.2 Discrete-time Markov chains 


3.2.1 Definitions and notations 


Definition 3.2.1. A stochastic process {Xy,n = 0,1,...} whose state space 
Sx,, is finite or countably infinite is a stationary (or time-homogeneous) 
Markov chain if 
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P|Xn+1 =Jj | Xn = 1,Xn-1 = In—1) cee , Xo = io] 
= PiXngi = j | Xa =i] = Pi (3.10) 


for all states i9,... ,in-1,1,7 and for any n > 0. 


Remarks. i) In the general case, we can denote the conditional probability 
P(Xn41 = J | Xn = å] by pi, j (n). Moreover, when there is no risk of confusion, 
we may also write pi j simply as pij. 

ii) When the states of the Markov chain are identified by a coding system, we 
will use, by convention, the set N° := {0,1,...} as the set space. For example, 
suppose that we wish to model the flow of a river as a Markov chain and that 
we use three adjectives to describe the flow X,, during the nth day of the year: 
low, average, or high, rather than considering the exact flow. In this case, we 
would denote the state low flow by state 0, the state average flow by 1, and 
the state high flow by 2. 


iii) Note that, in the example given above, the conditional probabilities 
P\Xn41 = j | Xn = i] actually depend on n, since the probability of moving 
from a low flow to an average flow, in particular, is not the same during the 
whole year. Indeed, this probability is assuredly smaller during the winter and 
higher in the spring. Therefore, we should, theoretically, use a nonstationary 
Markov chain. In practice, we can use a stationary Markov chain, but for a 
shorter time period, such as the one covering only the thawing period in the 
spring. 

In this book, we will consider only stationary Markov chains. Anyhow, 
the general case is not used much in practice. Moreover, in the absence of an 
indication to the contrary, we take for granted in the formulas and definitions 
that follow that the state space of the chain is the set {0,1,... }. 


Definition 3.2.2. The one-step transition probability matrix P of a 
Markov chain is given by 


0 1 2 
0 | Po0,0 Po,1 Po,2 --- 
1 | Pi,0 P1,1 P1,2 - 


P = 2] p20 p21 P22... (3-11) 


Remarks. i) We have indicated the possible states of the Markov chain to the 
left of and above the matrix, in order to facilitate the comprehension of this 
transition matrix. The state to the left is the one in which the process is at 
time n, and the state above that in which the process will be at time n + 1. 


ii) Since the p;,;’s are (conditional) probabilities, we have 


Pij > 0 V ae) l (3.12) 
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Moreover, because the process must be in one and only one state at time n+1, 
we may write that 


oe) 
X pi =] V2 (3.13) 
j=0 


A matrix that possesses these two properties is said to be stochastic. The sum 
ya Dis: for its part, may take any nonnegative value. If we also have 


OO 
X Pij =1 Vj (3.14) 
i=0 


the matrix P is called doubly stochastic. We will see, in Subsection 3.2.3, that 
for such a matrix the limiting probabilities, in the case when the state space 
is finite, are obtained without our having to do any calculations. 


We now wish to generalize the transition matrix P by considering the case 
when the process moves from state i to state j in n steps. We then introduce 
the following notation. 


Notation. The probability of moving from state i to state j in n steps (or 
transitions) is denoted by 


py SP Ami m] Amm i rna aU (3.15) 


From the p™’ 


ij S we can construct the matrix P™) of the transition prob- 
abilities in n steps. This matrix and P have the same dimensions. Moreover, 
we find that we can obtain P™® by raising the transition matrix P to the 


power n. 


Proposition 3.2.1 (Chapman—Kolmogorov equations). We have 
oo 
pet = Spl pi for m,n, i,j 2 0 (3.16) 
k=0 


Proof. Since we consider only stationary Markov chains, we can write, using 
the total probability rule, that 


DO = PX pan 9 | Xo=4 (3.17) 
OO 

= 50 PiXmin =5,Xm =k| Xo =7i (3.18) 
k=0 


We then deduce from the formula (see p. 74) 


P[ANB|C] = P[A | BNC]P[B|C] (3.19) 
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that 
ee i > PiXmin =j|Xm=k, Xo = iP | Xm =k | Xo [=i] 
k=0 
OO oO 
Sb lee, 0 (3.20) 


In matricial form, the various equations (3.16) are written as follows: 


pimtn) — pimpi (3.21) 
which implies that 
P™) = pph... pP® (3.22) 
e—a 
nx 


Since P™ = P, we indeed have 


p™ = p” (3.23) 


Example 3.2.1. Suppose that the Markov chain {Xn,n = 0,1,...}, whose 
state space is the set {0,1,2}, has the (one-step) transition matrix 


0 1 2 
01/3 1/3 1/3 
P=1i1] 0 1/21/2 
211 0 0 


We have 
4/9 5/18 5/18 
P? = P? = | 1/2 1/4 1/4 
1/3 1/3 1/3 


Thus, py? = 4/9. Note that to obtain this result, it is sufficient to know the 


first row and the first column of the matrix P. Similarly, if we are looking for 
py, we can use the matrix P|?) to calculate the first row of P“). We obtain 


139/324 185/648 185/648 
pM — pt = as — — 


Next, we have 


$) 139 1 185 185 833 
PE eee ETE anes 
Poo = 354% 3+ gag Tra a 
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So, in general, to obtain only the element pi” of the matrix P™® , it suffices 


to calculate the ith row of the matrix PTY? and to multiply it by the jth 
column of P. 

In Subsection 3.2.3, we will be interested in finding (if it exists) the limit 
of the matrix P‘) when n tends to infinity. To do so, we will give a theorem 
that enables us to calculate the limiting probabilities of the Markov chain. 
By making use of a mathematical software package, we can also multiply 
the matrix P by itself a sufficient number of times to see to what matrix it 
converges. Here, we find that if n is sufficiently large, then 


0.4286 0.2857 0.2857 
P) = P” ~ | 0.4286 0.2857 0.2857 
0.4286 0.2857 0.2857 


Observe that the three rows of the matrix above are identical, from which 
we deduce that whatever the state in which the process is at the initial time, 
there is a probability of approximately 0.4286 that it will be in state 0 after a 
very large number of transitions (this probability was equal to ~ 0.4285 after 
only five transitions, from state 0). By using the theorem of Subsection 3.2.3, 
we can show that the exact probability that the process is in state 0 when it 
is in equilibrium is equal to 3/7. 


Sometimes, rather than being interested in the transition probabilities in 
n steps of the Markov chain {X,,n = 0,1,...}, we want to calculate the 
probability that the chain will move to state j, from Xo = 12, for the first time 
at time n. 


Notation. The probability of moving to state j, from the initial state t, for 
the first time at the nth transition is denoted by 


os) = PIXn =f Xn-1#j,---, Xi Zil Xosi] forn>1 andi,j 20 
(3.24) 


Remarks. i) When i = J, o is the probability of first return to the initial 


state i after exactly n transitions. 


ii) We have the following relation between py”) and py: 


k) (n-k) 
pij = -5 Pry Pay (3.25) 


When the pls are known, we can use the preceding formula recursively to 
obtain the ott Ms for k=l 2.3 


Example 3.2.2. In some cases, it is easy to calculate directly the probabilities 
of first return in n transitions. For example, let 
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0 


1 
_ 0f1/3 2/3 
pat 


be the matrix of transition probabilities in one step of a Markov chain whose 


possible values are 0 and 1. We find that 


= (1/3)"""(2/8) = = 


ps?) = PiXn = 1, Xn-1 =0,... , X1 = 0 | Xo = 0] = 
for n = 1,2,... . Similarly, 
pi =1 and py) =0 for n = 2,3,... 


Next, we have 
poo = = 1/3, py) = = (2/3) x 1 = 2/3, and pa s0 forn=3,4,... 
() 0, pC) = = 2/3, and 


Finally, we calculate Pii = 











n 2 
pi") = 1 x (1/3)"-2(2/3) = goa frn=3,4.. 
Note that, in this example, we have } L4 pf? = aLI en py) = = 
OO oo 
(n) 27 1 _ 
Do a ee ae 
n=1 n=1 3 31-3 
and m 7 
(n) 2 2 2 2 1 _ 
2a oa aaa E 


n=l 
These results hold true because, here, whatever the initial state, the process 


is certain to eventually visit the other state 


Example 3.2.3. In Example 3.2.1, we calculated P). We also find that the 


matrix P®) is given by 
0 1 2 

0 [ 23/54 31/108 31/108 

5/12 7/24 7/24 


p® = 3 
4/9 5/18 5/18 


2 


From this, we deduce from Eq. (3.25) that, for instance 
(1) 


l 1) (1-1 
= Pb = PoAPia =l => Po. = 3 


3 
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and 
5 1)_(Q-1 2) (2-2 
og = POL = POAPLa + PO1PLa 
1 { 
Enoe E We 
Fo 2 
and 
3l 3 1 (2) (3-2 3)_(3~3) 
soe Pol = Poi Pi eo er nee fe ere oe 
a oe ee ee (3) 4 
=gxgtgxgtruxt =F Poi 57 
and so on. 


As in the preceding example, we can, at least for relatively small values of 
n, try to calculate directly re E, . Indeed, we have 


2 1 1 
Po. = Poa = 3) Pot = Poo x Poa = 


and 


1 1 1) 1 1 1 l 4 
P0.1 = Poo X Po,o X Po,1 + Poa X P3,0 X Pha = 5 + 5 = 57 


However, in general, the technique that consists in summing up the proba- 
bilities of all the paths leading from i to j in exactly n transitions rapidly 
becomes of little use. 


Until now, we have considered only conditional probabilities that the pro- 
cess {X,,n =0,1,...} finds itself in a state j at a time instant n, given that 
Xo = i. To be able to calculate the marginal probability P[X, = j], we need 
to know the probability that Xo = i, for every possible state 2. 


Definition 3.2.3. The initial distribution of a Markov chain is the set 
{a;i =0,1,...}, where a; is defined by 


Remarks. i) In many applications, the initial position is deterministic. For 
example, we often suppose that Xp = 0. In this case, the initial distribution 
of the Markov chain becomes ag = 1 and a; = 0, for i = 1,2,.... In general, 
we have J) oai = 1. 

ii) The marginal probability a P A = h orn = d; 2,244 1s obtained 
by conditioning on the initial state: 


ay” = PIX = So PIX n =j | Xo = jP 1= Yofa (3.27) 


420 4=0 


When ap = 1 for some k, we simply have a = oe 
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Example 3.2.4. Suppose that a; = 1/3, for i = 0,1, and 2, in Example 3.2.1. 
Then, we may write that 


C ae 
“0 ~3\9°2°3) 54 


CY Pires E ma 
oe =a =3( +773) > to 


and 


It follows that 


23 31 31 31 
Ba Woe tlk a eX aaa ap 
and 
31 31 155 
P ae 2 oa 2 Bay aie 
E|X5|=0+1 x ig t? XT 08 
so that 





155 /31\" 899 
v=- (3) ~ 1296 


Particular cases 


1) A (classic) random walk is a Markov chain whose state space is the set 
{... ,—2,—1,0,1,2,...} of all integers, and for which 


Pi, i+1 = P = 1— Pi i—i for 4 = Q, El +2. oe (3.28) 
for some p €E (0,1). Its one-step transition matrix is therefore (with q := 1—p): 


i—i ¿i i+li+2 


p= i (3.29) 


Thus, in the case of a classic random walk, the process can make a tran- 
sition from a given state i only to one or the other of its two immediate 
neighbors: 7 — 1 or i + 1. Moreover, the length of the displacement is always 
the same, i.e., one unit. If the state space is finite, say the set {0,1,...,N}, 
then these properties can be checked for each interior state: 1,2,...,N — 1. 
When the process reaches state 0 or state N, there are many possibilities. For 
example, if | 


Poo = PNN = 1 (3.30) 
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we say that the states 0 and N are absorbing. If po, = 1, the state 0 is said 
to be reflecting. 


Remarks. i) If p12 = p, pix = cg, and pio = (1 — c)q, where 0 < c < 1, the 
boundary at 0, when poo = 1, is said to be elastic. 


ii) We can also say that the process has a reflecting boundary at the point 
1/2 if the state space is {1,2,...}, p11 =q, and pi. = p. 


2) We can easily construct a Markov chain from a set Yọ, Y1,... of id. 
random variables, whose possible values are integers, by proceeding as follows: 
let | 


Xn ee for n =0,1,... (3.31) 
k=0 
Then, {X,,n = 0,1,...} is a Markov chain and, if 
kta oa Oe Nk (3.32) 


we have 


n+l 


Pee 
k=0 


SS PYner = j — i] = aji 


n 


SOY =i 


k=0 


Dig Pnt ae SP (3.33) 











=P yes 


k=0 





Yn41=j-1 








We say that the chain, in addition to being time-homogeneous, is also homo- 
geneous with respect to the state variable, because the probability pi,; does 
not depend explicitly on 7 and j, but only on the difference 7 — 2. 


3.2.2 Properties 


Generally, our first task in the study of a particular Markov chain is to de- 
termine whether, from an arbitrary state i, it is possible that the process 
eventually reaches any state j or, rather, some states are inaccessible from 
state 2. 


Definition 3.2.4. We say that the state j is accessible from i if there exists 
ann > 0 such that pe” > 0. We write i = 7. 
Remark. Since we include n = 0 in the definition and since p® = 1 > 0, any 


g~ 
state is accessible from itself. 


Definition 3.2.5. If i is accessible from j, and j is accessible from i, we say 
that the states i and j communicate, or that they are communicating. We 
write i e 9. 
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Remark. Any state communicates with itself. Moreover, the notion of com- 
munication is commutative: i + j = j |e i. Finally, since (by the Chapman- 
Kolmogorov equations) 


pe > py Dy for all m and n > 0 (3.34) 


we can assert that this notion is also transitive: if i e j and 7 e k, then 
i e k. Actually, it is sufficient to notice that if it is possible to move from 
state 7 to state 7 in m steps, and from j to k in n steps, then it is possible 
(by the Markov property) to go from i to k in m +n steps. 


Definition 3.2.6. Ifi ++ j, we say that the states 1 and j are in the same 
class. 


Remark. The transitivity property (see above) implies that two arbitrary 
classes are either identical or disjoint. Indeed, suppose that the state space is 
the set {0,1,2} and that the chain has two classes: {0,1} and {1,2}. Since 
0 > 1 and 1 © 2, we have that 0 + 2, which contradicts the fact that the 
states 0 and 2 are not in the same class. Consequently, we can decompose the 
state space into disjoint classes. 


Definition 3.2.7. Let C be a subset of the state space of a Markov chain. We 
say that C is a closed set if, from any i € C, the process always remains in 
C: 


P| Xn EC | Xn =iEeCl=1 foralliec (3.35) 


Definition 3.2.8. A Markov chain is said to be irreducible if all the states 
communicate, that is, if the state space contains no closed set apart from the 
set of all states. 


The following result is easy to show. 


Proposition 3.2.2. Ifi — 7 or j — i for all pairs of states i and 7 of the 
Markov chain {Xn, n = 0,1,...}, then the chain is irreducible. 


Remark. We can also give the following irreducibility criterion: if there exists 
a path, whose probability is strictly positive, which starts from any state i 
and returns to i after having visited at least once all other states of the chain, 
then the chain is irreducible. We can say that there exists a cycle with strictly 
positive probability. 


Example 3.2.5. The Markov chain whose one-step transition matrix P is given 
in Example 3.2.1 is irreducible, because we may write that 
| 1 


x x a eae 
Po,1 X P1,2 P2,0 = 3 5 -g 
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Note that here the cycle 0 — 1 — 2 — 0 is the shortest possible. However, we 
also have, for example, 


Po,2 X 2,0 X Po,1 X P1,2 X P2, > 0 
Remark. If a Markov chain (with at least two states) has an absorbing state, 
then it cannot be irreducible, since any absorbing state constitutes a closed 


set by itself. 


Definition 3.2.9. The state i is said to be recurrent if 


Jia e 








(J {Xn = i} 


n=1 


Xo = =1 (3.36) 


If fii <1, we say that i is a transient state. 


Remarks. i) The quantity fi; is the probability of an eventual return of the 
process to the initial state 7. It is a particular case of 


n=1 


aa _— P Xo = i (3.37) 








which denotes the probability that, starting from state 7, the process will 
eventually visit state 7. We may write that 


oO 
ee (3.38) 
n=l 


ii) It can be shown that the state space Sx, of a Markov chain can be decom- 
posed in a unique way as follows: 


Sx, =DUC, UC) UC}... (3.39) 


where D is the set of transient states of the chain and the C;’s, k = 1,2,..., 
are closed and irreducible sets (that is, the states of each of these sets com- 
municate) of recurrent states. 


Proposition 3.2.3. Let N; be the number of times that state i will be visited, 
given that Xo = i. The state i is recurrent if and only if E[N;] = œ. 


Proof. We have 
PiINg= nl =f sfa) fori tie: (3.40) 


That is, N; ~ Geom(p := 1 — fii). Since E[N;] = 1/p, we indeed have 
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1 


E[N;] = are 


=o <— fii=l 0D (3.41) 


Remark. Since the set T = {0,1,...} is countably infinite, that is, time con- 
tinues forever, if the probability of revisiting the initial state is equal to 1, 
then this state will be visited an infinite number of times (because the process 
starts anew every time the initial state is visited), so that PIN; = oo] = 1, 
which directly implies that E[N;] = oo. Conversely, if 7 is transient, then we 
have i 


P[N; = œ] = lim P[N; > n) = lim fyt =0 (3.42) 
n00 n= ? 

That is, a transient state will be visited only a finite number of times. Conse- 

quently, if the state space of a Markov chain is finite, then at least one state 

must be recurrent (otherwise no states would be visited after a finite time). 


Proposition 3.2.4. The state i is recurrent if and only if Yro pr = OO. 


Proof. Let Ii x„=i} be the indicator variable of the event {Xn = i}. That is, 


_ flifX, =1 
HXn=i} = {0 if X, #1 oe 
We only have to use the preceding proposition and notice that, given that 
Xo = i, the random variable I;x,,=;; has a Bernoulli distribution with pa- 
rameter p := P[X, = i | Xo = i], so that 








OO CO 
EIN] = E | So Tex, =| Xo = =) Elltx,=1 | Xo =í] 
n=0 n=O 
CO oO 
-Samin o a. 
n=0 n=0 


Remark. It can also be shown that if i is transient, then 


OO 
` pl” <oo for all states k 
n=1 


The next result is actually a corollary of the preceding proposition. 


Proposition 3.2.5. Recurrence is a class property: if states i and j commu- 
nicate, then they are either both recurrent or both transient. 
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Proof. It is sufficient to show that if 7 is recurrent, then j too is recurrent 
(the other result being simply the contrapositive of this assertion). 
ae definition of communicating states, there exist integers k and | such 


that p; E) 5 0 and p. 


3 > 0. Now, we have 


(l k l k 
aes er ye (3.45) 
Then 
l+nt+k (l 
daly yO 2 PhP Spl) = 00 (3.46) 
a es 
= 00 


Thus, j is recurrent. O 


Corollary 3.2.1. All states of a finite and irreducible Markov chain are re- 
current. 


Proof. This follows indeed from the preceding proposition and from the fact 
that a finite Markov chain must have at least one recurrent state (see p. 88). 
| p 


Notation. Let i be a recurrent state. We denote by u; the average number 
of transitions needed by the process, starting from state i, to return to 7 for 
the first time. That is, 


v= Dom ie (3.47) 


Definition 3.2.10. Let i be a recurrent state. We say that i is 


positive recurrent if H; < œ 
null recurrent if Hi = œ 


Remarks. i) It can be oe (see Prop. 3.2.4) that ¢ is a null recurrent state 
if and only if >, pi i) = 00, but ia pe = = 0. We then also have 
Dikss pi. = 0, for all states k. 

ii) It can be shown as well that two recurrent states that are in the same 


class are both of the same type: either both positive recurrent or both null 
recurrent. Thus, the type of recurrence is also a class property. 


iii) Finally, it is easy to accept the fact that any recurrent state of a finite 
Markov chain is positive recurrent. 
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Example 3.2.6. Because the Markov chain considered in Example 3.2.1 is finite 
and irreducible, we can at once assert that the three states—0O, 1, and 2—are 
positive recurrent. However, the chain whose one-step transition matrix is 
given by 
0 1 2 
0| 1/3 1/3 1/3 
2; 0 1 0 


is not irreducible, because state 0 is not accessible from either 1 or 2. Since 
there is a probability of 2/3 that the process will leave state 0 on its very first 
transition and will never return to that state, we have 


fo,o < 1 — 2/3 = 1/3 <1 (actually, foo = 1/3) 


which implies that state 0 is transient. Next, 


1 
P1,2 X P21 = 5 >0 


so that states 1 and 2 are in the same class. Because the chain must have at 
least one recurrent state, we conclude that 1 and 2 are recurrent states. We 
may write that Sx, = D U C4, where D = {0} and C, = {1,2}. 
When 
0 i 2 
0f 1/3 1/3 1/3 
P2 = 1| 0 1/21/2 
2; 0 O 1 
we find that each state constitutes a class. The classes {0} and {1} are tran- 


sient, whereas {2} is recurrent. These results are easy to check by using Propo- 
sition 3.2.4. We have 








mo prl =e 
2 Pon D2 3 1-4 a 
DUCED dat =2<00 
n=0 n=0 2 


and 
OO oO 
do P22 = D1 = 00 
n=0 n=0 


As we already mentioned, any absorbing state (like state 2 here) trivially 
constitutes a recurrent class. Moreover, the state space being finite, state 2 
is positive recurrent. Finally, we have that Sy, = D U C1, where D = {0,1} 
and Ci = {2}. 
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Definition 3.2.11. A state i is said to be periodic with period d if py = 0 
for any n that is not divisible by d, where d is the largest integer with this 
property. If d = 1, the state is called aperiodic. 

Remarks. i) If pk? = 0 for all n > 0, we consider 7 as an aperiodic state. That 
is, the chain may start from state i, but it leaves i on its first transition and 
never returns to 7. For example, state 0 in the transition matrix 


ri: 2 
0f01/21/2 
P = 1101/21/2 
2/0 0 1 


is such that Me = 0 for alln > 0. 

ii) As in the case of the other characteristics of the states of a Markov chain, 
it can be shown that periodicity is a class property: if i and 7 communicate 
and if 2 is periodic with period d, then j too is periodic with period d. 


iti) If pe? > 0, then 7 is evidently aperiodic. Consequently, for an irreducible 
Markov chain, if there is at least one positive element on the diagonal (from 
poo) of the transition matrix P, then the chain is aperiodic, by the preceding 
remark. 


iv) If p?) > 0 and pi? > 0, then 7 is aperiodic, because for any n € {2,3,...} 
there exist integers kı and ko € {0,1,...} such that n = 2k, + 3k2. 


v) If d = 4 for an arbitrary state i, then d = 2 too satisfies the definition of 
periodicity. Indeed, we then have 


po S0 Tora = 0 is 
n) 


so that we can assert that py = 0 for any n that is not divisible by 2. Thus, 
we must really take the largest integer with the property in question. 


Example 3.2.7. If 
01 


0,01 
P | 1 
then the Markov chain is periodic with period d = 2, because it is irreducible 
and 


por) =0 and pew) = 1 forn = 0, Less: 


Note that d is not equal to 4, in particular because 2 is not divisible by 4 and 


pop =1>0. 
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Example 3.2.8. Let 
0 1 2 
0| 0 1/21/2 
P=1/1/2 0 1/2 
2| 1/21/2 0 
The Markov chain is aperiodic. Indeed, it is irreducible (since po 1 X p1,2 X P2,0 = 
1/8 > 0) and 
Dyn (= 1/2) >0 and pi) (=1/4) >0 


Example 3.2.9. Let 


012 3 4 
0f01/21/2 0 0 
1}0 0 0 1/32/3 

P=2|0 0 0 2/31/3 
311 0 0 0 0 
411 0 0 0 0 


be the one-step transition matrix of a Markov chain whose state space is the 
set {0,1,2,3,4}. We have 


Po,1 X P1,3 X P3,0 X Po,2 X p2,4 X paso > O 


Therefore, the chain is irreducible. Moreover, we find that 


(n)  Jlifn=0,3,6,9,... 
Po,0 = ) 0 otherwise 


Thus, we may conclude that the chain is periodic with period d = 3. 


Example 3.2.10. In the case of a classic random walk, defined on all the inte- 
gers, all the states communicate, which follows directly from the fact that the 
process cannot jump over a neighboring state and that it is unconstrained, 
that is, there are no boundaries (absorbing or else). The chain being irre- 
ducible, the states are either all recurrent or all transient. We consider state 
0. Since the process moves exactly one unit to the right or to the left at each 
transition, it is clear that if it starts from 0, then it cannot be at 0 after an 
uneven number of transitions. That is, 


pea” =0 forn=0,1,... 


Next, for the process to be back to the initial state 0 after 2n transitions, there 
must have been exactly n transitions to the right (and thus n to the left). Since 
the transitions are independent and the probability that the process moves to 
the right is always equal to p, we may write that, for n = 1,2,..., 


poo = P[B(2n,p) = n] = Cra ap Cnr rq pe 
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To determine whether state 0 (and therefore the chain) is recurrent, we 


consider the sum 
=. (n) _ \>_(2n) 
> Po E S Poo 
n=l n=1 


However, we do not need to know the exact value of this sum, but rather 
we need only know whether it converges or not. Consequently, we can use 
Stirling’? formula: 


n! ~ ntie Von (3.48) 


(that is, the ratio of both terms tends to 1 when n tends to infinity) to 
determine the behavior of the infinite sum. We find that 
[4p(1 — p)|" 


af HTN 


2 
Pho) ~ 
which implies that 


OO OO 

4n(l — p)” 
Vp < 00 = y Ér p)] a 
n= 1 


Jan 


Now, if p = 1/2, the stochastic process is called a symmetric random walk 
and the sum becomes 


n=l 





n=l 
because = = 
1 11 
sein ppan 
2, AN St 2, n a 
When p # 1/2, we have 
OO oO 
= n es n 
[4p(1 — p)] D pa-p)" 
TN T 
n=l n= 


Thus, the classic random walk is recurrent if p = 1/2 and is transient if 
ps 1/2. 

Remark. It can be shown that the two-dimensional symmetric random walk 
too is recurrent. However, those of dimension k > 3 are transient. This follows 
from the fact that the probability of returning to the origin in 2n transitions 
is bounded by c/n*/?, where c is a constant, and 


oO 
> aE < ifk >3 (3.49) 
n=l 


2 James Stirling, 1692-1770, who was born and died in Scotland, was a mathemati- 
cian who worked especially in the field of differential calculus, particularly on the 
gamma function. 
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3.2.3 Limiting probabilities 


In this section, we will present a theorem that enables us to calculate, if it 
exists, the limit lim,_, pi”) by solving a system of linear equations, rather 
than by trying to obtain directly limno P™, which is generally difficult. 
The theorem in question is valid when the chain is irreducible and ergodic, as 
defined below. 


Definition 3.2.12. Positive recurrent and aperiodic states are called ergodic 
states. 


Theorem 3.2.1. In the case of an irreducible and ergodic Markov chain, the 
limit 


E ee), 
Tj := lim pij (3.50) 
exists and is independent ofi. Moreover, we have 
1 
a N for all j € {0,1,...} (3.51) 
j 


where u; is defined in Eq. (3.47). To obtain the m;’s, we can solve the system 


ae (3.52) 
OO 
Sony =1 (3.53) 
j=0 


where T := (To, T1,...). It can be shown that the preceding system possesses 
a unique positive solution. 


Remarks. i) For a given state j, Eq. (3.52) becomes 


OO 
t=) Tipaj (3.54) 
i=0 
Note that, if the state space is finite and comprises k states, then there are k 
equations like the following one: 


k-1 
Tj = > Ti Pi,j (3.55) 
i=0 


With the condition 2a T; = 1, there are thus k + 1 equations and k un- 
knowns. In practice, we drop one of the k equations given by (3.55) and make 
use of the condition PE T; = 1 to obtain a unique solution. Theoretically, 
we should make sure that the 7,’s obtained satisfy the equation that was 
dropped. 
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ii) A solution {7;,7 = 0,1,...} of the system (3.52), (3.53) is called a station- 
ary distribution of the Markov chain. This terminology stems from the fact 
that if we set 


P| Xo =j] =r; Vi (3.56) 


then, proceeding by mathematical induction, we may write that 


P[Xnt1 == > PiXnai =| Xn = i)P[X, = 4] 
1=0 


OO 
i=0 


where the last equality follows from Eq. (3.54). Thus, we find that 


P[X, =j} =a; Vn,j € {0,1,2,...} (3.58) 


iii) If the chain is not irreducible, then there can be many stationary distri- 
butions. For example, let 
O1 


0/10 
P= ton 


Because states 0 and 1 are absorbing, they are positive recurrent and aperi- 
odic. However, the chain has two classes: {0} and {1}. The system that must 
be solved to obtain the 7;’s is 


To = To (3.59) 
Wy = T1 (3.60) 
m+ =l (3.61) 


We see that 79 = c, 7; = 1 — c is a valid solution for any c € [0,1]. 


iv) The theorem asserts that the 7;’s exist and can be obtained by solving 
the system (3.52), (3.53). If we assume that the 7,’s exist, then it is easy to 
show that they must satisfy Eq. (3.52). Indeed, since limpoo P[Xn4i1 = j] = 
limMpoo P|Xn = j], we have 


OO 
P| Xn = j| = > ab ere =j|Xn=iP[Xn = i] 
~=0 


= lim P[Xn41 =f] = lim X 0 P[Xngi1 = 5 | Xn =i]P[Xn = i 
Nn OO Tt OO 


i=0 


OO OO 
me AE > | Pig Ti = Son Pij (3.62) 
i=0 


i=0 
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where we assumed that we can interchange the limit and the summation. 


v) In addition to being the limiting probabilities, the 7;’s also represent the 
proportion of time that the process spends in state j, over a long period of 
time. Actually, if the chain is positive recurrent but periodic, then this is the 
only interpretation of the 7;’s. Note that, on average, the process spends one 
time unit in state j for p; time units, from which we deduce that mj = 1/45, 
as stated in the theorem. 


vi) When the Markov chain can be decomposed into subchains, we can ap- 
ply the theorem to each of these subchains. For example, suppose that the 
transition matrix of the chain is 


0 1 2 
O11 0 0 
P=] 1) 01/2 1/2 
2/01/21/2 
In this case, the limit lim, pe” exists but is not independent of 7. We easily 
find that 
oe T ifi 40 oe) 
and (by symmetry) 
- (mo s(n) _ f 1/2ift=1or2 
a = 0 ifi=0 a) 


However, we cannot write that 7 = 1 and 7, = m2 = 1/2, since the 7,’s do 
not exist in this example. At any rate, we see that the sum of the 7;’s would 
then be equal to 2, and not to 1, as required. 


Example 3.2.11. The Markov chain {X,,n = 0,1,...} considered in Exam- 
ple 3.2.1 is irreducible and positive recurrent (see Example 3.2.6). Since the 
probability poo (or p1,1) is strictly positive, the chain is aperiodic, and thus 
ergodic, so that Theorem 3.2.1 applies. We must solve the system 


1/3 1/3 1/3 
(7,71, 72) = (To, 71, 02) 0 1/2 1/2 
1 0 QO 


that is, 


| 


1) 
To = zno + TM 


a 
NO 
Nee” 
m= wj m 


Ti eretan 


Ww 
Mo 


3) 1 i 
T2 = zro + zT 


Ww 
bo 
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subject to the condition a) + 7, + T2 = 1. Often, we try to express every 
limiting probability in terms of 79, and then we make use of the condition (*) 
to evaluate 7. Here, Eq. (1) implies that m2 = 270, and Eq. (2) enables us 
to assert that 7, = £0 too [so that mı = 72, which actually follows directly 
from Eqs. (2) and (3)]. Substituting into Eq. (*), we obtain: 
2 2 3 

To + zno + zmo =l => m= 7% > Mi = = 

We can check that these values of the 7,’s are also a solution of Eq. (3), which 


was not used. 


NIN 


Note finally that the 7,;’s correspond to the limits of the py 


in Example 3.2.1 by finding the approximate value of the matrix P\) when 
n is large. 


s computed 


When the state space Sx, is finite, before trying to solve the system (3.52), 
(3.53), it is recommended to check whether the chain is doubly stochastic (see 
p. 79), as can be seen in the following proposition. 


Proposition 3.2.6. In the case of an irreducible and aperiodic Markov chain 
whose state space is the finite set {0,1,... ,k}, if the chain is doubly stochastic, 
then the limiting probabilities exist and are given by 


1 
mT; =—— forj=0,1,...,k 3.65 
Proof. Since the number of states is finite, the chain is positive recurrent. 
Moreover, as the chain is aperiodic (by assumption), it follows that it is er- 
godic. Thus, we can assert that the 7;’s exist and are the unique positive 
solution of 


k 
1 = > Mi Dig V9 640,100 5k} (3.66) 
i=0 
k 
ET 
Now, if we set 7; = 1/(k + 1) > 0, we obtain 
sd 
D (3.68) 
—“k+i1 
j=0 
and 
k k 
1 i 1 1 
———_p; ,; = —— ga X |] = — 3.69 
Do pTI kei oe k+1 k+1 ae) 


(because the chain is doubly stochastic). 0 
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Example 3.2.12. The matrix 


0 1 2 
0] 1/3 1/3 1/3 
Ptr o t2173 
2} 2/3 1/6 1/6 
is irreducible and aperiodic (because po,o > 0, in particular). Since it is doubly 
stochastic, we can conclude that x; = 1/3 for each of the three possible states. 
We can, of course, check these results by applying Theorem 3.2.1. 


Example 3.2.13. The one-step transition matrix 


01 2 

of1 0 o0 

P = 1| 0 1/2 1/2 
2|01/21/2 


is doubly stochastic. However, the chain is not irreducible. Consequently, we 


cannot write that 7; = 1/3. Actually, we calculated the limits limn—oo p in 
remark vi) on p. 96. 


When the state space {0,1,...} is infinite, the calculation of the limiting 
probabilities is generally difficult. However, when the transition matrix is such 
that po > 0, Pj j-1 X Pj j+1 > 0Y j > 0, and 


pij; =O if|j-i|>1 (3.70) 
we can give a general formula for the 7;’s. Indeed, we have 
To = Topo,o + T1P1,0 (3.71) 
and 
| nj = Rj-1Pj-1,j + TjPjj + Tj+Pj+Lj (3.72) 
for j = 1,2,.... We find that 


_ Po, X P12 * +++ X Pj-1,7 


= to Tori = 1, 2,005 (3.73) 
Pi,0 X P2,1 X ++} X Pj j—-1 


Nj 
Then a necessary and sufficient condition for the existence of the 7,’s is that 


OO 
p x Kee K = 
0,1 P1,2 Pk-1,k < (3.74) 


ba P10 X P2,1 X tt X Pkik-1 


If the sum above diverges, then we cannot have that Da mj = 1. 
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Note that this type of Markov chain includes the random walks on the 
set {0,1,...}, for which po) = p = 1 — poo and piiga = p = 1 — Pi,i—1, for 
i = 1,2,.... We have 


OO CO 
Poi X Pi,2 Xt X Pk—-ik 5 


(3.75) 
koi EO Ee AD k=l 


ae 

— p)k 
E=) 
and this sum converges if and only if p < 1/2. Thus, we can write (with 
q := 1 — p) that 


CO 


TO S p/a)" =l e mals 4 (3.76) 
k=0 4 
and then 
j J 
1; = (2) To = (2) (1 = P) (3.77) 
q q q 
forga 17. sax 


Actually, we could have used Theorem 3.2.1 to obtain these results. Indeed, 
consider the transition matrix 


O i 2 e EEE kE-—-2k—-1k 
Q q p 
ı {q0 p 
2 q0 p 
P=: Big. Gee g (3.78) 
k—1 q 0 p 
k q p 


As the Markov chain is irreducible and ergodic, we can try to solve the system 


To =q To +q 71 
Nj = p nj—-1 Fq Ti+ FOE. 9 = date ,k-1 


Tk = P Te-1 tP Tk 


We find that 


p j 
n= (2) ie “TOR Sleek (3.79) 
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Then, the condition ae T; = 1 enables us to write 


n= (1-2) f : opi a 


Taking the limit as k tends to infinity, we obtain (in the case when p < q & 


p < 1/2) 
To > (1 = 2) and 7; > (2) (: - z) (3.81) 


for 7 = 1,2,..., which effectively corresponds to the formulas (3.76) and 
(3.77). 


3.2.4 Absorption problems 


We already mentioned (see p. 87) that the state space of a Markov chain can 
be decomposed into the set D of transient states of the chain and the union 
of closed and irreducible sets Ck of recurrent states. We are interested in the 
problem of determining the probability that, starting from an element of D, 
the process will remain indefinitely in D or instead will enter one of the sets 
Ck, from where it cannot escape. 


Notation. Let i € D and let C be a recurrent class. We set 


ry (C) = PIX, €C|Xo=i]= > pi) forn=1,2,... (3.82) 
jEC 


and 





L] {Xn € CH Xo = i (3.83) 


n=1 


ri(C) = lim r® (C) = P 





We have the following result. 


Theorem 3.2.2. The probability r;(C) is the smallest nonnegative solution 
of the system 


ri(C)= X pijri(C)+X pij for allie D (3.84) 
gED jEC 
Moreover, if D is a finite set, then the solution is unique. 
Remarks. i) The system (3.84) is a system of nonhomogeneous linear equa- 


tions. When D is finite, we can try to solve this system by using results from 
linear algebra. 
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ii) Actually, it is not necessary that the set C be a class. It is sufficient that 
C be a closed set of recurrent states. 


iii) We can imagine that all the states of the recurrent class C constitute a 
single absorbing state, since, once the process has entered this class, it cannot 
leave it again. 


A classic example of an absorption problem is known as the gambler’s 
ruin problem and is described as follows: a player, at each play of a game, 
wins one unit (for example, one dollar) with probability p and loses one unit 
with probability q := 1— p. Assume that he initially possesses 7 units and that 
he plays independent repetitions of the game until his fortune reaches k units 
or he goes broke. 

Let Xn be the fortune of the player at time n (that is, after n plays). Then 
{Xn n = 0,1,...} is a Markov chain whose state space is the set {0,1,... , k}. 
As states 0 and k are absorbing, we have that po,o = pp,k = 1. For all the other 
states i = 1,... ,k — 1, we may write that 


Pi i41 = P = t= Piai (3.85) 


This is a random walk on the set {0,1,... ,k}, with absorbing boundaries at 
0 and k. The chain thus has three classes: {0}, {k}, and {1,2,... ,k—1}. The 
first two are recurrent, because 0 and k are absorbing, whereas the third one 
is transient. Indeed, we have, in particular, that 


P[X; =0|Xop=Y=q>0 => Jia (3.86) 


from which we can conclude that the player’s fortune will reach 0 or k units 
after a finite number of repetitions. 

Let r;({0}), for i = 0,1,...,k, be the probability that the player will be 
ruined, given that his initial fortune is equal to i units. That is, we write that 
C is the class {0} in Eq. (3.83). We will first consider the case when p = 1/2. 
Note that, in this case, we have 


EX |Xo=i)=(i-1) x 5+ (it1)x =i fori=l...,b-1 (387) 
We also have 
E|X,| Xo =0]=0 and E[X,| Xo =k] =k (3.88) 
That is, in general, 
PAX par | Agi) =i foranyi (3.89) 


This type of Markov chain is a martingale and is very important for the 
applications, notably in financial mathematics. 
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Definition 3.2.13. A Markov chain for which 


E[Xn41 | Xn = i] =E ys J Pij = for all i E Sx, (3.90) 
JCOX;, 


is called a martingale. 


Remark. We can rewrite Eq. (3.90) as follows: 
Ba eX, (3.91) 


Now, proceeding by induction, we may write that 


i 
i = E[X, | Xo =} =X jp (3.92) 


Indeed, if we make the induction assumption that E[Xn-1 | Xo = i] = i, then 
we have 


k k k 
E[Xn | Xo = i] = odes = So api i 


j=0 1=0 
k k k | 
= 2? 22 ue = X pitE[Xn-1 | Xo = |] 
k 2 
= So pial = EX | Xo =i] =i (3.93) 
l=0 
Now, we have 
Jim | pe = =0Q for all transient states 7 (3.94) 


(otherwise the sum ae pr) would diverge, contradicting the remark on 


p. 88). It follows, taking the limit as n tends to infinity in Eq. (3.92), that 


i= lim OD; 9 D +kpik =k lim pix (3.95) 
j= 
That is, 
lim piy = z (3.96) 


from which we deduce that 


E -1_2 | eel 97 
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Thus, the more ambitious the player is, the greater is the risk that he will be 
ruined. 


In the general case where p € (0,1), the probability r;({0}), which will be 
denoted by r; to simplify the formulas, is such that ro = 1, rg = 0 and [see 
Eq. (3.84)| 


ry =pretq (3.98) 
Ti =qTfi-1 +PTri41 fori=2,...,k-—2 (3.99) 

and 
Tk-1 = qTk-2 (3.100) 


Rewriting Eq. (3.99) as follows: 
(p +q) ri =Qri-1 + Prii (3.101) 
we obtain 


fiji- T= 5 Te Pe) e ia n R (3.102) 


which implies that 


E TE D Cee (2) Cen (3.103) 


where the last equality follows from Eq. (3.98). 
Next, since r = 0, Eq. (3.100) can be rewritten as follows: 


Tk-1 = QTk-2 + PTk (3.104) 


We can then state that Eq. (3.103) is valid for i = 0,...,4 —1. Adding the 
equations for each of these values of i, from 0 to j — 1, we find that 


rj — ro = (rı —1) X (a/p)' (3.105) 
i=0 


Since ro = 1, we obtain 


lZ UB eina 


1+ j(r1 —1) ii p=¢ 


for 7 = 0,1,... , k. Finally, the fact that rẹ = 0 enables us to obtain an explicit 
expression for rı — 1 from the preceding formula, from which we find that 
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,— Le(a/py 


1 — (q/p)* nea 


(3.107) 
1-5 ifp=q 


Note that for p = q = 1/2, we retrieve the formula (3.97). 
We can calculate the limit of the probability r; in the case when k tends 
to infinity. It is easy to check that 


l (q/p) if p > 1/2 (3.108) 


BE ipa 


k— oo 
Thus, if p > 1/2, there exists a strictly positive probability that the process 
will spend an infinite time in the set of transient states. 


Another example for which the Markov chain may spend an infinite time 
in the set D of transient states of the chain is the following. 


Example 3.2.14. Let po = 1 and 
pio =a; (> 0) =1—p; 541 for j =1,2,... 


be the one-step transition probabilities of a Markov chain whose state space 
is the set {0,1,...}. This chain has two classes: {0} (recurrent) and {1,2,...} 
(transient). We calculate directly 


OO 


r)({0}) =1- ] [0 - aj) 


i=0 


It can be shown that r;({0}) < 1 if and only if the sum 5°; aj44 converges. 
For example, if a; = (1/2)*, for all i > 1, then we have 


= sat . . . 
S az = S (1/2) = (1/2) < cx 
7=0 4=0 


3.2.5 Branching processes 


In 19th-century England, some people got interested in the possibility that 
certain family names (particularly names of aristocratic families) would dis- 
appear, for lack of male descendants. Galton? formulated the problem math- 
ematically in 1873, and he and Watson‘ published a paper on this subject in 


* Francis Galton, 1822-1911, was born and died in England. He was a cousin of 
Charles Darwin. After having studied mathematics, he became an explorer and 
anthropologist. 

t The Reverend Henry William Watson, 1827-1903, was born and died in England. 
He was a mathematician who wrote many books on various subjects. He became 
a priest in 1858. 
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1874. Because Bienaymé° had worked on this type of problem previously, the 
corresponding stochastic processes are sometimes called (branching) processes 
of Brenaymé-—Galton—Watson. More simply, the term branching processes is 
also used. 


Definition 3.2.14. Let {Zn j n =0,1,... 39 =1,2,...} be a set of i.1.d. ran- 
dom variables whose possible values are nonnegative integers. That is, Sz; C 
{0,1,...}. A branching process is a Markov chain {Xy,n = 0,1,...} de- 
fined by 


Xn-1 
Xn = 2 Zai Xn- > 0 (3.109) 
j=1 
0 Wf any =O 
FOR WS 12) aay 


Remarks. i) In the case of the application to the problem of the disappearance 
of family names, we can interpret the random variables Xn and Zp—1,; as 
follows: Xo is the number of members of the initial generation, that is, the 
number of ancestors of the population. Often, we assume that Xo = 1, so that 
we are interested in a lineage. Z,_1,; denotes the number of descendants of 
the jth member of the (n — 1)st generation. 


ii) Let 
pi := P(Z_-1,; = 1] for all n and j (3.110) 


To avoid trivial cases, we assume that p; is strictly smaller than 1, for all 
t = 0,1,.... We also assume that pọ > 0; otherwise, the problem of the 
disappearance of family names would not exist. 

The state space Sx, of the Markov chain {X,,n = 0,1,...} is the set 
{0,1,...}. As state 0 is absorbing, we can decompose Sx,, into two sets: 


Sx, = DU {0} (3.111) 


where D = {1,2,...} is the set of transient states. Indeed, since we assumed 
that po > 0, we may write that 


P[X, #iV n€ {1,2,...} | Xo =i] > pio == pi > 0 (3.112) 


Thus, fi; < 1, and all the states i = 1,2,... are effectively transient. Now, 
given that a transient state is visited only a finite number of times, we can 


5 Irénée-Jules Bienaymé, 1796-1878, was born and died in France. He studied at 
the Ecole Polytechnique de Paris. In 1848, he was named professor of probability 
at the Sorbonne. A friend of Chebyshev, he translated Chebyshev’s works from 
Russian into French. 
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assert that the process cannot remain indefinitely in the set {1,2,...,k}, for 
any finite k. Thus, we conclude that the population will disappear or that its 
size will tend to infinity. 

Suppose that Xo = 1. Let’s now calculate the average number jin of indi- 
viduals in the nth generation, for n = 1,2,... . Note that uw; = E[X;] is the 
average number of descendants of an individual, in general. We have 


Mn = E[Xn] = >| E[Xn | Xn—1 = AP[Xn-1 = J] 


wo 
(>) 


juiıP|Xn-1 =] = p E|Xn-1] (3.113) 


Me: 


J 


li 
© 


Using this result recurrently, we obtain 
ln = Wi E[Xp-1] = ME|Xn-2] =... = wPE[Xo] = u? (3.114) 


Remark. Let o? := V[X,]. When u, = 1, from Eq. (1.96), which enables us 
to express the variance of Xn in terms of E[X, | Xn-1] and of VLXp | Xn-1], 
we may write that 


V[Xa] = ELV[Xn | Xn-i]] + VIE[Xn | Xn-1]] 
a E{Xn-107] at V[Xn~1 x 1] 
= o? x 1+ ViXn-1] = 207 F V[Xn-2] 


=... = no? + V[Xo] = no? (3.115) 


since V[Xo] = 0 (Xo being a constant). When u; Æ 1, we find that 





n] 
Vixa] = oup (E) (3.116) 
Hı 


We wish to determine the probability of eventual extinction of the popu- 
lation, namely, 


doe lim PL, = 0) Xo t] (3.117) 
TUF OO 
By independence, we may write that 


q0,i = 9.1 (3.118) 


Consequently, it suffices to calculate go,1, which will be denoted simply by qo. 
We have 


P[X, =0| Xo =1] =1-P[X, >1|Xo=1=1- >) P[Xn =k | Xo= YJ 
k=1 
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s15 bP, eset 

a oe (3.119) 
[by Eq. (3.114)]. It follows that, if u; € (0,1), then 

qo > lim 1-pP=1 = g=! (3.120) 


which is a rather obvious result, since if each individual has less than one 
descendant, on average, we indeed expect the population to disappear. 

When f4; > 1, Eq. (3.119) implies only that gg > 0 (if wi = 1) or that 
go > —œ (if yı > 1). However, the following theorem can be proved. 


Theorem 3.2.3. The probability qo of eventual extinction of the population 
is equal to 1 if pı <1, while qo < 1 if p> 1. 


Remarks. i) We deduce from the theorem that a necessary condition for the 
probability gg to be smaller than 1 is that p; must be greater than 0 for at 
least one j > 2. Indeed, if pọ = p > 0 and pı = 1 — p, then we directly have 
fy =1-p<il. 

ii) When po = 0 and pı = 1, we have that zy = 1. According to the theorem, 
we should have go = 1. Yet, if pj = 1, it is obvious that the size of the 
population will always remain equal to Xo, and then gg = 0. However, the 
theorem applies only when pp > 0. 


Let F be the event defined by 
FE k= oy (3.121) 
n=1 


so that qo = P[F | Xo = 1]. To obtain the value of qo, we can solve the 
following equation: 


OO (0.0) 
qo = X PIF | X1 = jl p; = X kp; (3.122) 
j=0 j=0 


The equation above possesses many solutions. It can be shown that when 
hı > 1, go is the smallest positive solution of the equation. 


Remark. Note that gg = 1 is always a solution of Eq. (3.122). 


Example 3.2.15. Suppose that pọ = 1/3 and p; = 2/9, for j = 1,2,3. First, 
we calculate 


2 4 
m=0+e(1+24+3)=5>1 


Thus, we may assert that gg < 1. Eq. (3.122) becomes 
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1 2 7 3 
go =z +zlo +g +) => gtag-5%+5=0 
3 9 2 2 
Since gg = 1 is a solution of this equation, we find that 


| 3 
(qo — 1) (<8 +200 - 5 = 0 


The three solutions of the equation are 1 and —1 + \/5/2. Therefore, we 
conclude that gg = —1 + ./5/2 ~ 0.5811. 
Remark. If pọ = 1/2 and p; = 1/6, for j = 1, 2,3, then we have 


1 
hi Aa aa 1 
Consequently, Theorem 3.2.3 implies that gg = 1. Eq. (3.122) is now 


1 1 
qo = 5 + ghd +9 + 90) <> (go—1)*(g0+3) =0 


so that the three solutions of the equation are 1 (double root) and —3, which 
confirms the fact that go = 1. 
Similarly, if pọ = pı = 1/2, then we find that 


1 
qo = 3 (1 + 4) <> =l 
in accordance with Theorem 3.2.3, since wy = 1/2 < 1. 


Assume again that Xo = 1. Let 
Ta:= Y X,=1+5_ Xk (3.123) 
k=0 k=1 


That is, Ta designates the total number of descendants of the population’s 
ancestor, in addition to the ancestor himself. It can be shown that 


YP (lim T,, = jl Ssi (3.124) 
j=l 
Thus, if qo < 1, then Ta := limyno Tn is a random variable called defective, 
which takes the value oo with probability 1 — qo. Moreover, if gg < 1, the 
mathematical expectation of Tæ is evidently infinite. When gg = 1, we have 
the following result. 


Proposition 3.2.7. The mathematical expectation of the random variable Too 
is given by 





1 
ElTol= zy fms! (3.125) 
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Remark. We observe that when u = 1, we have that P|Tæ < œ] = 1, but 
Ela = 00. 


Example 3.2.16. If po = p € (0,1) and pı = 1 — p, then we know that qo = 1, 
because u; = 1 — p < 1. In this case, we may write that 


x, — 1 with probability (1 — p)" 
k 10 with probability 1 — (1 — p)" 


Thus, we have 
n 
ETa =1+ġ 0 -p "2 —— = - 
k=1 


which is indeed equal to 1/(1 — u1). 


3.3 Continuous-time Markov chains 


3.3.1 Exponential and gamma distributions 


In the case of discrete-time Markov chains, we said nothing about the time 
the processes spend in state i before making a transition to some state 7. As 
in Example 2.1.1 on random walks, we may assume that this time is determin- 
istic and is equal to one unit. On the other hand, an essential characteristic of 
continuous-time Markov chains is that the time that the processes spend in a 
given state has an exponential distribution, so that this time is random. More- 
over, we will show that the sum of independent exponential random variables 
(having the same parameter) is a variable having a gamma distribution. We 
already mentioned the exponential and gamma distributions in Chapter 1. In 
the present section, we give the main properties of these two distributions. 


Exponential distribution 


Definition 3.3.1. (Reminder) Jf the probability density function of the con- 
tinuous random variable X, whose set of possible values is the interval |0, œœ), 
is of the form 


_ eTe? if x > 0 
fx (z) =f 0 frei (3.126) 


we say that X has an exponential distribution with parameter à > 0 and 
we write X ~ Exp(à). 


Remarks. i) Using the Heaviside function u(x) (see p. 11), we may write that 


fx(x) = àe™™u(z) (Vx ER) (3.127) 
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Note that we have the following relation between the functions u(x) and 6(x) 
(see p. 63): 


Ue) = f ô(t) dt (3.128) 


ii) For some authors, a random variable X having an exponential distribution 
with parameter A possesses the density function 


Paos se? Au(z) (3.129) 


The advantage of this choice is that we then have E[X] = A, while for us 
E[X] = 1/A, as will be shown further on. 


The distribution function of X is given by 


0 ifz<0 
_ £ 
Fx (x) = f je dt =1-e7™ ifz >0 (57180) 
0 
Note that we have the following very simple formula: 
P[X >z] =e forx>0 (3.131) 


Some people write F(x) for the probability P[X > a]. 

The main reason for which the exponential distribution is used so much, in 
particular in reliability and in the theory of queues, is the fact that it possesses 
the memoryless property, as we show below. 


Proposition 3.3.1. (Memoryless property) Suppose that X ~ Exp(A). 
We have 


PIX >s+t|X>t]=P[X >s] Vs,t2>0 (3.132) 


Proof. By the formula (3.131), we may write that 


PIX >s+t,X >t] PIX >s+tt 
Pix >t] — PIX >f] 

e7 A(stt) 
o Àt 


P|X >s+t|X >t] = 
=e% = PX >s) O (3.133) 


Remark. Actually, the exponential random variables are the only r.v.s having 
this property for all nonnegative s and t. The geometric distribution possesses 
the memoryless property, but only for integer (and positive) values of s and t. 


We can easily calculate the moment-generating function of the r.v. X ~ 


Exp(\): 
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Mx (t) = Efe'*) = / i ee" dr = A A z ift<AÀ (3.134) 
Then 
Mx(t) = EET and My(t)= D T (3.135) 
which implies that 
E[X] = M% (0) = 5, E[X?] = M%(0) = = ad Vite 5 (3.136) 


Remark. Note that the mean and the standard deviation of X are equal. 
Consequently, if we seek a model for some data, the exponential distribution 
should only be considered if the mean and the standard deviation of the 
observations are approximately equal. Otherwise, we must transform the data, 
for example, by subtracting or by adding a constant to the raw data. 


Example 3.3.1. Suppose that the lifetime X of a car has an exponential dis- 
tribution with parameter A. What is the probability that a car having already 
reached its expected lifetime will function (in all) more than twice its expected 
lifetime? 


Solution. We seek 


2 1 1 
PX SS ay eee `| =e! ~ 0.3679 
> 5/x> 5] P|x>5| e 3 


Remark. The assumption that the lifetime of a car has an exponential distribu- 
tion is certainly not entirely realistic, since cars age. However, this assumption 
may be acceptable for a time period during which the (major) failure rate of 
cars is more or less constant, for example, during the first three years of use. 
Incidentally, most car manufacturers offer a three-year warranty. 


As the following proposition shows, the geometric distribution may be 
considered as the discrete version of the exponential distribution. 


Proposition 3.3.2. Let X ~ Exrp(à) and Y := int(X)+1, where “int” des- 
ignates the integer part. We have 


PS kale) Se). for k= 1 ya (3.137) 
That is, Y ~ Geom(p := 1—e74). 


Proof. First, since int(X) € {0,1,...}, we indeed have that Sy = {1,2,...}. 
We calculate 


P[Y =k] = Plint(X) = k- 1] = Pk -1< X <k] (3.138) 
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k 
zs / New rt e a E a ee | ane en) g 
—l 


Remark. If we define the geometric distribution by 
P\Y =k] =qfp for k=0,1,2,... (3.139) 


(as many authors do), then we simply have that Y := int(X) ~ Geom(p = 
1—e7). 


We can also consider the exponential distribution on the entire real line. 


Definition 3.3.2. Let X be a continuous random variable whose density func- 
tion ts given by 


Ix(z) = 2 zNa forxeR (3.140) 


where is a positive parameter. We say that X has a double exponential 
distribution or a Laplace® distribution. 


Remark. We find that the mean value of X is equal to zero and that V[X] = 
2/A?. The fact that E[X] = 0 follows from the symmetry of the function fx 
about the origin (and from the existence of this mathematical expectation). 


Proposition 3.3.3. If X ~ Exp{X), then for all xo = 0, we have 


E(X | X > zo] = zo + E[X] and VIX |X >zọļ=V[X] (3-141) 


Proof. The formula P[X > zo] = e~**° implies that 
fx(x | X > zo) = Ae **-7) for x > zo (3.142) 


from which we have 


OO OO 
E[X | X > xo] = J q eT to) dr E f (y + ao)Ae7 *Y dy 
0 


Tü 


= E[X] + zoP[X € [0, 00)] = E[X] + x (3.143) 


Next, we calculate 


OQ Q 
E(x? | X> To] = i r? Ae (2-0) dr MEET RO J (y ji zo)? AeA dy 
0 


= EIX? + 2x9 E|X] + 22P[X € [0, 00)] 
= E[X?] + 2roE[X] + 22 (3.144) 


© See p. 19. 
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Then we obtain 


V[X | X > xo] = E[X? | X > xo] — (E[X | X > zo}? 
= BX I= (EX) =VX 0 (3.145) 
Remark. The preceding proposition actually follows directly from the memo- 
ryless property of the exponential distribution. 


A result that will be used many times in this book is given in the following 
proposition. 


Proposition 3.3.4. Let Xı ~ Erp(àı) and X2 ~ Exp(à2) be two independent 
random variables. We have 


Aj 
PIX, < Xə) = ———_ 3.146 
[Xi < Xa] = (3.146) 
Proof. By conditioning on the possible values of X1, we obtain 
OO 
Pas = f Ce Ee ee OTC (3.147) 
0 


d oO DO 
= / P[X2 > alAzye"*"* dz = / et ne“ de 
0 0 


a À 
2 Are Oit) ge = og 3.148 
/ 1€ T a T y ( ) 


Remarks. i) We have that P| X; < X2] = 1/2 if Ay = A2, which had to be the 
case, by symmetry and continuity. 


ii) The proposition may be rewritten as follows: 


At 
Pi Xi = mini X1, X2} = —-—— 3.149 
[X, = min{X1, X2} reer (3.149) 
Moreover, we can generalize the result: let X1,... , Xn be independent random 
variables, where X;, ~ Exp(A,), for all k. We have 
Piet ha (3.150) 
: aaa alee Cece eo 
To prove this formula, we can make use of the following proposition. 
Proposition 3.3.5. Let X, ~ Exp(\x), ..., Xn ~ Exp(An) be independent 
random variables, and let Y := min{X1,... , Xn}. The r.v. Y has an expo- 


nential distribution with parameter À := ài +... + An. 


Proof. Since min{X;, X2, X3} = min {X;, min{ X2, X3}}, it suffices to prove 
the result for n = 2. We have, for y > 0, 
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PIY > y] = P[X1 > y, X2 > y) "È P[X > y|P[X2 > yl 
— eT AY eT ÀY — eT AitAa)y 
= fy (y) = 1 + Age 424 u(y) (3.151) 


Example 3.3.2. Suppose that X,, Xo, and X3 are independent random vari- 
ables, all of which have an Exp(A) distribution. Calculate the probability 
P(X, < (X2 + X3)/2]. 


Solution. We have P |X: < (X2 + X3)/2] = P[Y < X2 + X3], where Y := 
2X1. Moreover, for y > 0, 


PIY < y] = PIX; <y/2j=1-e°4/2 = Y ~ Exp(à/2) 
Then, by the memoryless property of the exponential distribution, 


P[2X; < X2 + X3] = 1 — PLY > X2 + Xz! 
= 1 — PY > X2 + X3 | Y > X| P[Y > Xa] 





2 
d 
=1 — PIY > Xs|PIY > X} $ 1- 5 
| AAS 


= 1 — (2/3)? = 5/9 


Finally, we would like to have a two-dimensional version of the exponential 
distribution. We can, of course, define 


fxi xa(£1, £2) = àge 2814272) for z; > 0, 2 > 0 (3.152) 


However, the random variables X, and Xə are then independent. To ob- 
tain a nontrivial generalization of the exponential distribution to the two- 
dimensional case, we can write 


P\Xy > L1,X2 > T2] = exp{—A,21 = A2k2 = Ai2 max{Z 1, £2}} (3.153) 


for x1 > 0, x2 > 0, where 12 is a positive constant. We indeed find that X; ~ 
Exp(A,;), for i = 1, 2. ~ 


Another possibility is the random vector whose joint density function is 
(see Ref. [5]) 


(3.154) 





AiA2 A121 + A2z2 2(pr;Agax12)!/? 
fx, X2 (21, T2) = i. exp a oeo Ip a =" ae 


for xı > 0, x2 > 0, where p € [0, 1) is the correlation coefficient of X; and X23, 
and Io(-) is a modified Bessel’ function of the first kind (of order 0) defined 
by (see p. 375 of Ref. {1]) 


T Friedrich Wilhelm Bessel, 1784-1846, was born in Germany and died in Königs- 
berg, in Prussia (now Kaliningrad, in Russia). He was an astronomer and mathe- 
matician. The mathematical functions that he introduced in 1817 are important 
in applied mathematics, in physics, and in engineering. 
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Ape. 1,232 1,233 
OER GET GE 


Here, too, we find that X; ~ Exp(A,) and Xə ~ Exp(,2). 


ae (3.155) 








Gamma distribution 


Definition 3.3.3 (Reminder). We say that the continuous and nonnegative 
random variable X has a gamma distribution with parameters a > 0 and 
A> 0, and we write that X ~ Gla, à), if 
(Xn)? er" 
P(a) 


where I'(-) is the gamma function, defined (for a > 0) by 


fx(x) = u(x) (3.156) 


mas / to-le dt (3.157) 
0 


Remarks. i) The function I’(a) is strictly positive for any positive a. For 
a > 1, we have 


OO 
Tla) = lemt +(a- 1 | pee at 
0 
"2°04 (a-1)IF(a-1) =(a—1)F(a-1) (3.158) 
Then, since 
oO 
Th) = | e*dt=1 (3.159) 
0 
we have 


Pin) = (n- DLP (n— 1) = (n— 1)(n — 2)P(n — 2) 
=... =(n—I1)(n—-2)---1-F(1)=(n-1)! (3.160) 


Thus, the gamma function generalizes the factorial function. We also have 


I'(1/2) = [ utent it we V2e~* /? ds (3.161) 
z =i | -= ee ekaa PNO Da 0] = Va 


ii) Contrary to the exponential distribution, whose density function always has 
the same form, the shape of the density function fx changes with each value of 
the parameter a, which makes it a very useful model for the applications. We 
say that the parameter a is a shape parameter, while A is a scale parameter (see 
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Fig. 3.1. Examples of probability density functions of G(a@, A = 1) random variables. 


Fig. 3.1). In reality, the shape of the function fy varies mostly when a is small. 
When a increases, the gamma distribution tends to a Gaussian distribution, 
which follows from the fact that if X ~ G(n,A), then the random variable 
X can be represented as the sum of n independent r.v.s X, ~ Exp(A), for 
k =1,...,n (see Prop. 3.3.6). 


iii) If X ~ G(a = 1, A), we have 
{x(Z)= Ne ** u(x) (3.162) 


Thus, the gamma distribution generalizes the exponential distribution, since 
G(a = 1,A) = Exp()). 

iv) The parameter @ may take any real positive value. When a = n € N, 
the gamma distribution is also named the Erlang? distribution. Moreover, we 
have that G(a = n/2,A = 1/2) = x2. That is, the chi-square distribution 
with n degrees of freedom, which is very important in statistics, is a particular 
case of the gamma distribution, too. 


The moment-generating function of X ~ G(a, A) is 


px -a (NEO AS i (t-r)2a-1 
= tr’ = £ nA d 
Mx(t) / e Tla) dz raok e x £ 





y=(A-t)x A2 f ama AS 
AE E Ba Sane Cr 
Tape, A Y= Tear | 


3 Agner Krarup Erlang, 1878-1929, was born and died in Denmark. He was first 
educated by his father, who was a schoolmaster. He studied mathematics and nat- 
ural sciences at the University of Copenhagen and taught in schools for several 
years. After meeting the chief engineer for the Copenhagen telephone company, 
he joined this company in 1908. He then started to apply his knowledge of proba- 
bility theory to the resolution of problems related to telephone calls. He was also 
interested in mathematical tables. 
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À Q 

= — fi À 3.163 
(5 = :) ort< ( ) 

We then calculate 

+ Í 
E|X] = Mx(0) = and E[X?] = M¥(0) = seri (3.164) 
so that 
ala+1) /a\?_ a 

ah cee 3.165 
de acid Go Maar (3.165) 


We can transform a G(a,A) random variable X into an r.v. having a 
G(a,1) distribution, sometimes called the standard gamma distribution, by 
setting Y = AX. Indeed, we then have 


a—lo-Y 
frio) = ix A- Fay (3.166) 


The distribution function of Y can be expressed as follows: 





Fy(y) = aot for y > 0 (3.167) 


where (a, y) is the incomplete gamma function, defined by 


y 
van) = f pete dt (3.168) 
0 


We have the following formula (see p. 272 of Ref. {1]): 
y(a,y) =a ye” M(1,1 +a,y) (3.169) 


where M(-,-,:) is a confluent hypergeometric function, defined by (see p. 504 
of Ref. [1]) 


ala+1)z? aļla+1)(a+2)z’ 








M = — no ae 3.170 
(a,b,2) = 145 D * b(b+1) 1 B+ 1)(b+2) al nm) 
When a = n € N, the function (a, y) becomes 
R yk 
y(n,y) =IP(n) |1—-e74 > a (3.171) 
k=0 ` 
which implies that 
n—i yk 
Fy(y)=1- > ena for y > 0 (3.172) 


k=0 
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This formula can be rewritten as follows: 


PIY <yj=1-P[W<n-1]=P|W>n], where W ~ Poi(y) (3.173) 


Remarks. i) The formula (3.173) can be obtained by doing the integral 


[ t tet dt (= 4(n,y)) (3.174) 
0 


by parts (repeatedly). 


ii) We will see in Section 5.1 that, in the case of a Poisson process (with rate 
A = 1), the random variable Y represents the time needed for n events to 
occur, whereas W is the number of events that occur in the interval [0, y}. In 
other words, the relation between Y and W is expressed as follows: the nth 
event of the Poisson process occurs at the latest at time y if and only if there 
are at least n events in the interval [0, y]. 


Example 3.3.8. Suppose that the duration T (in hours) of a major power fail- 
ure is a random variable having a gamma distribution with parameters a = 
2 and À = 1/2 (so that the average duration is equal to four hours). What 
is the probability that an arbitrary (major) power failure lasts more than six 
hours? 


Solution. First, we have 
PIT > 6] = P[X > 3], where X ~ G(2,1) 
Then, by using the formula (3.173), we can write that 


P[X > 3] = P[Poi(3) < 1] = e-3(1 + 3) = 4e7* ~ 0.1991 


Remarks. i) It is important not to forget that T (or X) is a continuous random 
variable, while W is discrete. 


ii) When the parameter a is small, as in this example, we can simply integrate 
by parts: | 


PIX >3 | ee ee ad 
> = t= Te £ 
, TES), 


oO 
= -xe |z + e`? dr = 3e? +e™°? = 4e? 
3 
We have seen that the exponential distribution is a particular case of the 


gamma distribution. We will now show that the sum of independent exponen- 
tial random variables, with the same parameter, has a gamma distribution. 
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Proposition 3.3.6. Let X;,...,X,, be independent random variables. If Xi 
has an exponential distribution with parameter A, for all i, then 


i=1 
Proof. Let S := } `; Xi. We have 
exp f D Ai i ns I] Mx, (t) 
i=l i=l 


_ a 3.176 
I ae (S) fort <A ( ) 


¿=1 


Ms(t) = Ele] = E 





Since [A/(A—t)]” is the moment-generating function of an r.v. having a G(a = 
n, A) distribution, the result is then obtained by uniqueness. Indeed, only the 
G(a = n, A) distribution has this moment-generating function. O 


The exponential and gamma distributions are models that are widely used 
in reliability. Another continuous random variable, which also generalizes the 
exponential distribution and which is very commonly used in reliability and 
in many other applications, is the Weibull ® distribution. 


Definition 3.3.4. Let X be a continuous random variable whose probability 
density function is given by 


fx(z) = F(t) ew | (252) Jue (3.177) 








Ò 


We say that X has a Weibull distribution with parameters 8 > 0, yER, 
and 6 >Q. 


Remark. The exponential distribution is the particular case where 8 = 1, 
y = 0, and 6 = 1/). Like the gamma distribution, this distribution has a 
shape parameter, namely 8. The parameter y is a position parameter, while 
6 is a scale parameter. When y = 0 and 6 = 1, we have 


fx(x) = pafe u(z) (3.178) 


This variable is called the standard Weibull distribution. Its distribution func- 
tion is 

Fx(z)=1-e™ forz>0 (3.179) 
° E. H. Wallodi Weibull, 1887-1979, was born in Sweden and died in France. In 


addition to his scientific papers on the distribution that bears his name, he is the 
author of numerous papers on strength of materials, fatigue, and reliability. 
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Because the exponential, gamma, and Weibull distributions are very im- 
portant in reliability, we will end this section with the definition of the failure 
rate of a device, or a system, etc. 


Definition 3.3.5. The failure rate (or hazard rate) of a device, whose 
lifetime X is a continuous and nonnegative random variable, is defined by 
fx (t) 


rx(t) = iF (3.180) 


Remarks. i) We have that rx(t) ~ P[X e€ (t,t + dt] | X > t]/dt. That is, the 
failure rate rx (t), multiplied by dt, is approximately equal to the probability 
that a device being t time unit(s) old will fail in the interval (t,t + dt], given 
that it is functioning at time t. | 

ii) To each function rx(t) there corresponds one and only one distribution 
function Fx (t). Indeed, we have 


d 
rx(t) = fx(t) i s Fx(t) 


Inn (3.181) 


from which we can write (since Fy (0) = 0) that 


t t 4 Fyx(s) t = 
[ rxias= f aye) eh Fx (t)| 


=> Fy(t)= i — exp -f rx(s) ds} (3.182) 


Particular cases 
1) If X ~ Exp(A), then 


e7 ^t 


Note that this result is a consequence of the memoryless property of the expo- 
nential distribution. We say that the parameter A is the rate of the exponential 
distribution. 


2) In the case of the standard gamma distribution, we may write that 


pa-1,-t 


3) Finally, if X has a standard Weibull distribution, we deduce from Eqs. 
(3.178) and (3.179) that 
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rx() =p (3.185) 


In practice, the function rx(t) has more or less the shape of a bathtud. 
That is, at the beginning the failure rate decreases, then this rate is rather 
constant, and finally the failure rate increases (which is equally true for the 
death rate of humans). To obtain this kind of curve, we can consider a random 
variable that is a linear combination of three Weibull distributions: 


X := c1 Xi +2 Xo + C3 X3 (3.186) 


with c; > 0, for all i, and cı +c2+c3 = 1 (called a mized Weibull distribution), 
where X; has a 8 parameter smaller than 1, Xo a 8 parameter equal to 1, 
and X3 a @ parameter greater than 1. 


3.3.2 Continuous-time Markov chains 


Let {X (t), t 2 0} be a continuous-time and discrete-state Markovian [see Eq. 
(3.1)] stochastic process, and let 7; be the time that the process spends in 
state i before making a transition to some other state. We may write that 


Pitz >s+t|m>t] = Ply > s] ¥s,t>0 (3.187) 


Indeed, since the process is Markovian, the time it has spent in a given state 
does not influence the future. Consequently, whatever the time that the pro- 
cess has already spent in i, it is as likely that it will remain there during at 
least s additional time units than if it had just entered this state. Eq. (3.187) 
means that the continuous random variable 7; possesses the memoryless prop- 
erty. As we already mentioned (see p. 110), only the exponential distribution 
possesses this property. We can therefore conclude that 7; has an exponential 
distribution, with parameter denoted by v;, which, in general, depends on 
state t. 

Moreover, the Markov property also implies that the next state visited, 7, 
is independent of 7;. Thus, when the process leaves state i, it enters state j 
(# i) with probability p: ; (by definition), where 


OO 
pig=0 Vi and So pij=1 Vi (3.188) 
j=0 


The p;,;’s are the one-step transition probabilities of the embedded (or as- 
sociated) discrete-time Markov chain. Note, however, that, contrary to the 
transition matrices in the preceding section, all the terms on the main (de- 
creasing from top left to bottom right) diagonal of the matrix are necessarily 
equal to zero, by definition of the p;,;’s in the present case. 

The process {X (t), t > 0} is called a continuous-time Markov chain, which 
is now defined formally. 
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Definition 3.3.6. Let {X(t),t > 0} be a continuous-time stochastic pro- 
cess whose state space is N° = {0,1,...}. We say that {X(t),t > O} is a 
continuous-time Markov chain if 


PIXG +s) = 9). Xs) =] LA) =o.40 a7 <5) 
= PIX(t +) = j | X(s) = i] = pis) (3.189) 


Vs,t>0 andV i,j,z, E N®. 


Remarks. i) As in the case of the discrete-time Markov chains, we assume that 
the chains considered have stationary or time-homogeneous transition proba- 
bilities. We could treat the general case and denote the conditional probability 
P(X (t) = j | X(s) = i] by p;,;(s,t), where t > s, but since the most important 
processes for the applications are indeed such that p; ;(s,t) = pij(t — s), it 
will not be necessary. 


ii) Continuous-time Markov chains are also known as Markov jump processes. 


iii) The function p;,;(t) is called the transition function of the continuous-time 
Markov chain. 


iv) The probabilities p; ;(t) correspond to the pis in discrete-time Markov 
chains. If there exist a t > 0 for which p; ;(t) > 0 and a t* > 0 for which 
p;,i(t*) > 0, we say that states i and j communicate. The chain is irreducible 


if all states communicate. 


v) We may write that 
X pilt) =L Vi (3.190) 
j=0 


since the process must be in some state at time t + s, regardless of the state 
it was in at time s. 


vi) For the sake of simplicity, we will assume in the sequel that the state space 
of the Markov chain {X (t), t > 0} is, save indication to the contrary, the set 
{0,1,...}. However, as in the discrete case, the state space can actually be a 
set Sx) C {0,1,...}, or, more generally, a finite or countably infinite set of 
real numbers. 


Example 3.3.4. Since all the elements of the main diagonal of the matrix P 
in Example 3.2.8 are equal to zero, we can consider this matrix as the transi- 
tion matrix of the embedded discrete-time Markov chain of a continuous-time 
Markov chain. The fact that p;,; = 1/2, for all i 4 j, does not mean that all 
the random variables 7; have the same parameter v;, for i = 0,1, 2. 


The following proposition is the equivalent, in the continuous case, of 
Proposition 3.2.1. 
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Proposition 3.3.7 (Chapman—Kolmogorov equations (2)). For all s,t 
nonnegative, we have 


Pi k(t T s) = Last Dz kls = Laut S)D;, k(t) (3.191) 


Proof. The proof is similar to that of Proposition 3.2.1. O 


Example 3.3.5. In general, it is not easy to calculate the functions p; ;(t) ex- 
plicitly. T heorems 3.3.1 and 3.3.2 provide differential equations whose solu- 
tions are these p;,;(t)’s. In the case of a Poisson process (see Section 5.1), we 
find directly that 


ovat (At)? 
(7-1)! 

Pij (t) = 
0 forj—i<0 


for 7-1 >0 


where À is the rate of the process. We indeed have, for k > i, 


= sap (AHT as) 
2 Pis (pals) = = Spt )Pj,n(S) Ea i ae a Gen 


j=i 
ti-—tgk—j : ti gk -*-J 
= e7 Àlt+ts) \ ko a = = e Altts) \k-i 
Yoon k — j)! D ile a Hy 
k~i k-i-~ k-i 
= eo MES) et V - = Ten tisi (x) oo Mt+s) yki t tS) 5) 
N 5 JEN E 
k—i 
e-A(t+s) [A(t + lh _ i 


where Eq. (*) is obtained by Newton’s!? binomial theorem. 


Notation. We denote by p;(t) the (marginal) probability that the process 
{X(t),t > 0} will be in state j at time t: 


BOS Pix) = 3 (3.192) 
If a; := P[X(0) = i], for i=0,1,..., then we may write that 


OO 
p(t) = X a; pi j(t) (3.193) 
i=0 
When P[X(0) = k] = 1 for some k € {0,1,...}, we simply have that p,(t) = 
Pk, g(t). 
10 Sir Isaac Newton, 1643-1727, was born and died in England. Newton was a scholar 
who is famous for his contributions to the fields of mechanics, optics, and astron- 


omy. He is one of the inventors of differential calculus. He also wrote theological 
books. 
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3.3.3 Calculation of the transition function p;,;(t) 


First, if state i is absorbing, which means that the parameter v; of the random 
variable 7; is equal to 0, we may write that 


tikisa 
walt) = 0a 352 j 3.194 
Pij (t) = ĝi; : a ( ) 
for all t > 0. In the case of nonabsorbing states, we will obtain two systems 
of differential equations, which, when we can solve them, give us the value of 
pi j(t), for all states i, j € {0,1,...}. 
Remark. It can be shown that the p; ;(t)’s are continuous functions of t, for 
every pair (i,j). 
Definition 3.3.7. The quantities 

Vij (= Vi Pij V4 Æj = {0, Lya .} (3.195) 


are the infinitesimal parameters or instantaneous transition rates of 
the continuous-time Markov chain {X(t),t > 0}. 


Remark. We have 
` Wj Uy X pij =p; (because pj; = 0) (3.196) 
j#t jŁi 

We set 


Vii = —Vi (3.197) 


? 


It follows that 
S ran (3.198) 
= 


Definition 3.3.8. The matriz 


0 | Vo o Y0,1 40,2 --- 
1] V10 Y1,1 V12- 


G = 2| V20 V21 V22... 


(3.199) 


is known as the generating matrix of the continuous-time Markov chain 
{X (t), t 2 0}. 


Remarks. i) If we know the quantities v; j, for all i # j, then we can calculate 
the rates v; and the probabilities p;,; from Eq. (3.195). Moreover, we will show 
in this section that the p;,;(t)’s depend only on the v,’s and the p;,;’s, from 
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which one derives the name of generator of the Markov chain for the set of all 
Vij ’s. 

ii) The matrix G corresponds to the transition matrix P for a discrete-time 
Markov chain. However, note that the 1,,;’s are not probabilities and that the 
sum of the elements of each row of G is equal to 0 rather than to 1. 


Notation. If the function g(x) is such that 


then we write that g(x) = o(2). 


A function g(x) that “is” o(x) must therefore tend to 0 more rapidly than 
the identity function f(x) = « when x — 0. Thus, gi(z) := x? is o(x), while 
g2(x) := yz is not o(x). Moreover, if g;(x) = o(x), fori =1,...,n, then 


n 
XN cigi(z) =o(r) VqGEeR (3.201) 
¿=l 


Proposition 3.3.8. The probability that a continuous-time Markov chain, 
{X(t),t > 0}, makes two or more transitions in an interval of length 6 is 
ol). 


Proof. We know that the time 7; that the process spends in state i has an 
exponential distribution with parameter v;, for i = 0,1,.... Suppose first 


that v; = v V i and, without loss of generality, that X(0) = 0. Let N be the 
number of transitions of the Markov chain in the interval [0, ô]. We have 


PIN > 2) = 1 — PIN = 0] — PIN = 1] 


CO på 
=1- Pio >d] -Y / Gi iee Pie SS ail a 
ka 9 


Oo ò 
ais e v6 = ron | ye vu e`” (5u) du 
k=1 0 


lo g) 
IC X Pok e “yf E a ET (3.202) 


k=1 
Using the series expansion 
ae 
falt+eta+...=1lt+2+o(z) (3.203) 


we may write that 


PIN > 2} =1-e7”°(1 46) =1—[1 — vd + 0(5)|(1 +5) = 0(5) (3.204) 
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Now, if the v;’s are not all equal, it suffices to replace each v; by v := 
max{V,V1,...} to prove the result. Indeed, since E[7;] = 1/%, the larger v; 
is, the shorter the average time that the process spends in state 7 is. But, since 
for the largest v; we can assert that PIN > 2] = o(ô), this must hold true 
when y; <v Vi. D | 


Proposition 3.3.9. We may write that 
d mar. 
Vij = Pii (t) V BIE {0, 1, soii } (3.205) 
t t=0 


Proof. Suppose first that i = 7. We know, by the preceding proposition, that 
the probability of two or more transitions in an interval of length 6 is o(6). It 
follows that 


pi ilô) = P[ri > 6] + 0(5) = e™™? + 0(5) = 1 — 1,6 + o(ô) (3.206) 
Since p;;(0) = 1, we may write that 


pes(8) — Pes) _ 4,4 19 (3.207) 


Taking the limit on both sides as 6 decreases to 0, we obtain 


pi i(ô) — pi (0) = —rið + 0(6) => 


When i 4 j, we have that p; ;(0) = 0. Then 


pijl) = Pir; < 6] pij + 0(6) = (1 —e7”*) pij +o(8) = vi ô pij +olô) 


(3.209) 
so that 
. Bi,g(d) — pig (0) . o(d) ; 
p; ;(0) = mi E = Vi Dig + mn eg iy O (3.210) 


We now give the first system of differential equations that enables us to 
calculate the p;,;(t)’s. 


Theorem 3.3.1 (Kolmogorov backward equations). For all states i,j 
e N°, and for allt > 0, we have 


OO 
pi g(t) = X vir pr alt) (3.211) 
k=0 
Proof. We decompose the probability p;,;(t) into two incompatible cases: 


pi g(t) = Pin < t, X(t) = j | X(0) = i] + Pfr: >t, X(t) = j | X(0) = i] 
(3.212) 
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Since P[r; > t| = e~”**, we may write that 
Ply >t, X(t) = j | X(0) = i] = et ôi; (3.213) 
Next, we have 
t 
Ply < t, X(t) = j | X(0) =i = | ie" Y pis Pk (t — s))ds (3.214) 
0 kżi | 
so that 


t 
Di, 3 (t) = echt 04 + / pie "s ( Y Pik Pkg (t = s) ds (3.215) 
0 


ki 


Remark. This integral equation allows us to state that the p;,;(t)’s are continu- 
ous functions of t, as we already mentioned above. It follows that the function 
that we integrate is a continuous function as well. 


Finally, to obtain the Kolmogorov backward equations, it suffices to first 
rewrite Eq. (3.215) as follows: 


t 
Pij (t) = e"it Si; + pe Vit f pit ( Y Pik pryl) du (3.216) 
o kżi 
and then to differentiate this last equation: 


t 
Pi, (t) = -nfet de vie Vit | eit ( X Pik Pk,j (u)) du} 
0 


kži 
sige ter `^ Pi,k Pk i(t) 


kži 
= - pi j(t) + vi Y Pik Ping (t) = Vviapi jlt) +X vik pr (t) 
r Fi 
OO 
=Ý %kPrg(t) O (3.217) 


k=0 


Remark. If we set t = 0 in the Kolmogorov backward equations, we obtain 


OO 
pi (0) = XO vie Pig (0) = vij Pj (0) = vij (3.218) 
k=0 


which confirms the result in Proposition 3.3.9. Actually, from Proposition 
3.3.9, we can show directly the validity of the Kolmogorov backward equations. 
Indeed, we deduce from the Chapman—Kolmogorov equations that 
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d Ad 
TzPis(s +t) = =D grl s) pk jt) V s,t > 0 (3.219) 


=> pi,(t)= S Ope (t) (with s = 0) 
k=0 


=> pilt) =X vik Pe,j(t) 
k=0 
where we assumed, in the first equation, that we can interchange the derivative 
and the summation. 


The second system of differential equations that we can use to calculate 
the p; ;(t)s is valid when, as in the remark above, we can interchange the 
derivative and the summation when we differentiate the Chapman—Kolmogo- 
rov equations. Now, this interchange is allowed when the state space of the 
Markov chain is finite and, also, in particular, for the birth and death processes, 
which will be defined in the next subsection. 


Theorem 3.3.2 (Kolmogorov forward equations). Under the condition 
mentioned above, for all i,j € N°, and for any t > 0, we have 


OO 
pi jlt) = S > pin (t) Vk, j (3.220) 
Proof. We have, under the condition in question, 


d = d 
—— 7): >: = , —— ° 5 t > 0 3.221 
dees (t + s) > Di,k(t) dge kd (s) Vs,t2 ( ) 


OO 
if s= 
Soph j(t) = Y pielt) ph (0) 


oo 
= pilt) =X pirlt) Vej 


by Proposition 3.3.9. 0 


Remarks. i) Since there is a Kolmogorov equation for every pair (i,j), the 
two systems of differential equations comprise m? equations each, where m 
is the number of elements in the state space of the Markov chain {X(t), 
t > 0} (which can be infinite). Thus, obtaining an explicit solution for the 
probabilities p; ;(t) from these systems of equations is generally very difficult. 
In the next subsection, we will calculate the p;,;(t)’s when the state space has 
only two elements. 


ii) In certain particular cases, for example, in the case of the Poisson pro- 
cess, we can determine the p,,;(t)’s without having to solve the Kolmogorov 
differential equations. 
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Example 3.3.6. Suppose that the Markov chain {X(t),t > 0} has state space 
{0,1,...}, and that the process, from state 0, can only move to state 1. That 
is, po,ı = 1. We then have 


— Vo ifk =0 
k= Ve ik= 
0 otherwise 


It follows that the Kolmogorov backward equation for the pair (0,7) is given 
by 
Po,j (t) = volpi i(t) — pos (t)] 


If state 0 is absorbing instead, we have the following trivial result: 
Poo(t)=1 and poj(t)=0 for any j £0 


Note that since the two functions are constant, the differential equation above, 
which becomes pp ;(t) = 0 (since vo = 0), is satisfied. Conversely, the solution 
of the differential equation po (t) = 0 is po,;(t) = c, and the fact that po,;(0) = 
doj, that is, the initial condition, enables us to determine the value of the 
constant c. 


3.3.4 Particular processes 


A continuous-time, two-state Markov chain 


The first particular case that we consider is that for which the state space of 
the continuous-time Markov chain {X(t),t > 0} is the set {0,1}. We assume 
that these states are not absorbing, so that they communicate. Since the 
process, from state 0 (respectively, 1), can only move to state 1 (resp., 0), we 
have that po, = pi = 1. The generating matrix of the chain is given by 


lt wy ri 


G = P a | (3.222) 


from which we deduce that the four Kolmogorov backward equations are, for 
any t > 0, 


Poot) = —vo po,o(t) + vo p1,0(t) 

po (t) 2 —v po,i(t) + vo P1,1 (t) (3.223) 
piot) = n poolt) — “1 pi olt) 

pi (t) 2 Vy po,i(t) — vi pi, 1(t) 


Moreover, we have the following initial conditions: 


pij(0) = 5;; for i,j =0,1 (3.224) 
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Since 


Po,o(t) + po,i(t) = pi o(t) + p1,1(t) = 1 (3.225) 


it is sufficient to consider Eqs. (1) and (3) of the system (3.223). We deduce 
from these equations that 


Poolt) + p3 o(t) = (v1 — Yo) [Po,0(t) — P1,0(t)] (3.226) 
Thus, if vo = vı, we have 
Poo(t)+ Pi oft) =0 => poo(t)+piol(t) =e (3.227) 


a constant. The initial conditions (3.224) imply that c = 1. Eq. (1) can there- 
fore be rewritten as follows: 


Poolt) = —volpoo(t) — 1 + poo(t)] <=> — poo(t) + 20 po,o(t) = vo 


(3.228) 
Multiplying on both sides by e7”°', we may write that 
a y | 
dt (e*"" po o(t)) = vo e™ (3.229) 
The general solution of this ordinary differential equation is 
1 
e7¥F ng g(t) = so + co (3.230) 


where co is a constant. Making use of the condition po,9(0) = 1, we find that 
co = 1/2, so that 


1 
poolt) = 5 (1+ et) for t > 0 (3.231) 


From this function, we can calculate the other three functions p;,;(t). 


Remark. The general solution of the ordinary differential equation (o.d.e.) 
F’(rz) + c F(z) = G(x) (3.232) 


where c is a constant, is (for x > 0) 
F(z) = e7% (Fo) + | e*G(y) iy) (3.233) 
0 


When vo £ v1, subtracting Eq. (3) from Eq. (1) in the system (3.223), we 
find that 


Poolt) — Pio(t) = —(ro + vi )[po,o(t) — p1,0(t)] (3.234) 
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Using the preceding remark with F(t) := po o(t) — pio(t) and G(t) = 0, we 
may write that the solution of this o.d.e. is 


po,o(t) — pi,o(t) = {po,0(0) — pio(O)Je" Mott = e7 ott (3.235) 
Substituting into Eq. (1), we obtain 
ph olt) = -roe Metr (3.236) 


from which we deduce that 


: t 
/ Po o(s) ds = -| vga errs ds 
0 o” 0 
V 


0 
w + Vy 





<=> po,o(t) = po,o(0) — (2 - ee) (3.237) 


That is, 








= vı Vo — (+r )t 
= + ge roa”: Nt 0 3.238 
pooli) = i + 0 (3.238) 
Note that if vo = vı in this formula, then we retrieve the formula (3.231). 
Next, by using Eq. (3.235), we find that 


g Ki- -= t 
i= Dag Ot a ot e Wott YE>dO 


(3.239) 





Finally, Eq. (3.225) implies that 








ae — Vo Vo —(votv )t 
[k= 1 = H= haa €e 1 y t> 0 3.240 
Po,1( ) Po,o( ) i F 5 n = ( ) 


and 
Vo Vy 


a wera Y> (3.041) 
Yor wry 


pı (t) = 1 —pro(t) = 





As in the discrete case, an important problem (which will be treated in the 
next subsection) is that of determining, if it exists, the limiting probability 
lim¿—oo Pi,;(t). Here, we easily find that 


4 if} =O andi=0,1 
vo + vi 


jim Pi,j (t) = (3.242) 
pee | ae 
if j = 1 and ¿į = 0,1 








o + ly 


Note that the limit exists and does not depend on the initial state i. 
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Pure birth processes 


Definition 3.3.9. Let {X(t),t > 0} be a continuous-time Markov chain 
whose state space is the set {0,1,...}. If 


Pigi =l fort =U, l; (3.243) 
the process is called a pure birth process. 


Remarks. i) A pure birth process is thus a continuous-time Markov chain for 
which transitions can only be made from an arbitrary state to its right-hand 
neighbor. It is a particular case of the birth and death processes, which will be 
treated further on. 


ii) We can also define a pure birth process by setting 


Pij = i 0 otherwise (3.244) 


for alli Æ j € {0,1,...}. 
The Kolmogorov forward equations for a pure birth process are the fol- 
lowing: 


Pi i(t) = ~vi pilt) (3.245) 
and 
pi g(t) = 5-1 Pig—r(t) -vi pilt) ifj= itl it.. (3.246) 


Proposition 3.3.10. The function p;,;(t) for a pure birth process is given, 
for any t > 0 and for alli € {0,1,...}, by pij(t)=0 ifj <i, and 


eat Gi =1 


ig(t) = Pa ia oa 3.247 
P mil ) viest f ei p; j—1(8) ds if j >i ( ) 
0 


Proof. Since the process can move only to the right, we indeed have p; ;(t) = 0 
if 7 <i. 
Next, let F be the following event: the process makes no transitions in the 
interval [0, t]. Since the process cannot move backward, we may write that 
piilt) = P[F] = Pir, >t] = e7” (3.248) 


Finally, by (3.233), the solution of Eq. (3.246) is 


t 
pig(t) = e7" {Pia (0) + 5-1 | e”i p; 3_1(8) ds} (3.249) 
0 
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and the result follows from the fact that p; ;(0) = 0, for j =71+1,1+2,....0 


From the formulas (3.247), we can calculate p;;(¢) recursively. For in- 
stance, we have 


t 
Pii+1(t) = ernst f ev'47%p; (s) ds 
0 


t t 
= pemn | EFIS pm Vas ds = T. eVi+i—vi)s ds 
0 0 


Ute He ify; = Diri 
(3.250) 


Vi EES iy ; 
+ (et — eo") if v A Viya 
Vi+1i — Vi 


Particular cases 


1) Poisson process. This process is obtained by setting v; = À, for all 1 € 
{0,1,...}. To obtain its transition function p; (t), we can use Eqs. (3.247) 
and (3.250), with v; = 441 = A. We have 


e: A (At)? at 
Dewi) =A te™ = piinolt) = re™ | eiee ds = ae 
0 
(3.251) 
Proceeding recursively, we obtain the following general formula: 
otis CON eK = 
Piitk(t) = ee ford = Vand k= 0-1, x: (3.252) 
That is, 
parkt) = P[Poi(At) = k] (3.253) 


The parameter A is called the rate of the process. 


We can also obtain the preceding formula without making use of the Kol- 
mogorov equations, proceeding as follows: since v; = A, the random variables 
7; all have an exponential distribution with parameter A. Moreover, they are 
independent, by the Markov property. It follows, by Proposition 3.3.6, that 


S= Tit... + 744-1 ~ G(k,A) Vk 21 (3.254) 


Then, we may write that 





. = ' —: : Cl —rXs „—AÀ(t—s8) d 
Pii+klt) = | fs(s) Plri4n > t-— s] ds = 7 e “e S 
0 0 (k — )! 
PTEE (At)* 


t 
= k-1 _ p,—At 
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for i,k € {0,1,...} (since pi ;(t) eo) et) and Vt > 0. 


2) Yule! process. This process, studied by Yule in the framework of the 
theory of evolution, can be interpreted as follows: we consider a population 
whose members cannot die and such that each member, independently of the 
others, gives birth to descendants according to a Poisson process with rate 4. 
It follows that 


y,=ii Vi>0 (3.256) 


Indeed, when X(t) = i, the time 7; that the process spends in this state is 
given by _ 


Ti = min{X,,... ,X;} (3.257) 


where the random variables X1,... , X; are independent and all have an ex- 
ponential distribution with parameter A. By Proposition 3.3.5, we may write 
that r; ~ Exp(; = iA). 


Remark. Note that if X(0) = 0, then the process remains in this state forever. 


We can calculate the transition function of the process in various ways. 
We find that 


pig (t) = # 7 a ae —e “J iff>i>1 (3.258) 
To check this result, we use the fact that Eq. (3.246) becomes 
p(t) = (J — 1) pij-1(t)— FA pig (t) forg=i+1,i+2,... (3.259) 


Next, from the formula (3.258), we may write that 





j— i 1 


Moreover, we calculate [also from (3.258)] 


pi l(t) = iA pig (t) + (j — 1)Ae™™ pi j-1(t) (3.261) 
We have 
— ià pi s(t) + (j — LAN"; 5-1 (t) = (j — 1)A pi j-1(t) — JÀ piz (t) 
=> (j —1)(1-— e7™™) pi j-1(t) = (j — i) pi, j(t) (3.262) 


11 George Udny Yule, 1871-1951, was born in Scotland and died in England. He 
first obtained an engineering degree. Next, he became interested in the field of 
statistics and wrote important papers on regression and correlation theory. His 
book Introduction to the Theory of Statistics, published for the first time in 1911, 
was very successful. 
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which holds true by Eq. (3.260). 


Finally, the function given in (3.258) does indeed satisfy the initial condi- 
tion 


pi,j(0) = ôij (3.263) 
Remark. When i = 1, the formula (3.258) becomes 
prjt)=e™“1l-e™)-* iff >1 (3.264) 
We can then write that 
pij(t) = P[Geom(p := e™™*) =j] iff >1 (3.265) 


Now, when X(0) = 7 > 1, it is as if we added į independent random variables, 
Xj 1,...,X;, all of which have a geometric distribution with parameter p = 
e^t. We can show that such a sum S has a negative binomial or Pascal!” 
distribution with parameters 7 and p, whose probability mass function is given 
by 


Tl 


PÍS =j] = (: B 2 p(l—py* forj=i,it+1.,... (3.266) 


The formula for p; (t) thus follows directly from that for p;;(t). 


Birth and death processes 


Definition 3.3.10. If the instantaneous transition rates v; į of the continuous- 
time Markov chain {X(t),t > 0} are such that 


nj=0 if|j-if>1 (3.267) 
the process is said to be a birth and death process. 
Definition 3.3.11. The parameters 
Ai = hiiti (fori >0) and Hi:= nai- (fori >1) (3.268) 


are called, respectively, the birth and death rates of the birth and death 
process. 


12 Blaise Pascal, 1623-1662, was born and died in France. He is one of the founders 
of the theory of probability. He was also interested in geometry and in physics, 
in addition to publishing books on philosophy and on theology. 
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Remark. Other terms used to designate the parameters À; and ju; are the fol- 
lowing: growth or arrival rates, and mortality or departure rates, respectively. 


From the instantaneous transition rates Vi j, we can calculate the param- 
eters v; of the random variables 7;, as well as the probabilities p; j, for all 
states 7 and j in the set {0,1,...}. We have 


Vo = —V0,0 = %,1 = ro (3.269) 
and 
Vi = —Vii = Viii Vii- = Ai +p fie 1. (3.270) 
In the case when the state space is the finite set {0,1,...,m}, we have 
Dn = a Vena = im (3.271) 


If \; = u; = 0, then state 7 is absorbing. For all nonabsorbing states 
i € {0,1,...}, we have that po = 1 and, using the formula (3.195), 


Vi i+1 Ài nes 
Piji+1 v: Non: Pi,ji-1 2 ( ) 


Remarks. i) A birth and death process is a continuous-time Markov chain for 
which transitions, from state i, can only be made to i — 1 (if i > 0) ori +1. 
In a short interval, of length 6, the probability that the process moves from 
state i to i + 1 (respectively, i — 1) is equal to A;6 + 0(6) (resp., pô + o(ô)). 
Thus, 1 — (A; + ui)ô + 0(6) is the probability that the process will still be (or 
will be back) in state i after 6 unit(s) of time. 

ii) In many applications, the state of the process at a given time instant is 
the number of individuals in the system at this time instant. A birth and 
death process may then be interpreted as follows: when X(t) = 7, the waiting 
time until the next arrival is a random variable X; having an Exp(,;) distri- 
bution and which is independent of the waiting time Y;, having an Exp(j;) 
distribution, until the next departure. We then have To = Xo ~ Exp(% = Ao) 
and 


ti = min{X;,Y;} ~ Exp(, =A; + pi) fori >0 (3.273) 


Moreover, we indeed have poi = 1 and 





ind. Ài pecs 
Digi = PIX; < Yi] = P[Exp(\;) < Exp(:)] = x = ifi>O (3.274) 
2 t 


Similarly, 





Me ede 
Pit = > 3 ifi>0 (3.275) 


Hi 
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iii) A pure birth process is the particular case where the death rates u; are 
all equal to zero. A process for which A; = 0 for all i is called a pure death 
process. 


Particular cases 


1) The continuous-time Markov chain whose state space is the set {0,1}, 
considered at the beginning of this subsection (see p. 129), is an example of a 
birth and death process, for which 


Ao = vo and p = (3.276) 


(all the other birth and death rates being equal to zero). 


2) Suppose that we modify the Yule process as follows: after an exponential 
time with parameter A, an individual either gives birth to a descendant, with 
probability p, or disappears from the population, with probability 1 — p. Let 
X(t) be the number of individuals in the population at time t. The process 
{X(t),t > 0} is now a birth and death process whose birth and death rates 
are given by 


A; =tpA and pwj=i(l—p)rA forz>0 (3.277) 


If we assume rather that, when an individual gives birth to a descendant, 
there is a probability p (respectively, 1 — p) that this individual remains in 
(resp., disappears from) the population, then the process {X(t),# > 0} is no 
longer a continuous-time Markov chain. Indeed, suppose that X(0) = 1. We 
may write that 


Pin <t] = p(t ap PIN enc Np (3.278) 
k=1 


where the X;’s are independent random variables that all have an exponen- 
tial distribution with parameter à. This result follows from the fact that we 
perform independent trials for which the probability of success, that is, the 
case when the individual remains in the population, is equal to p. Since the 
random variable 7, does not have an exponential distribution, {X(t),t > 0} is 
not a continuous-time Markov chain anymore. Actually, 7, is an infinite linear 
combination of independent gamma distributions. 


3) The queueing model M/M/s. This process will be studied in detail 
in Section 6.3.1. We suppose that customers arrive at a system according 
to a Poisson process with rate å. There are s servers, but the customers 
form a single queue. The service times are independent and have an Exp(j) 
distribution. We have 


A =A Vi>0, w=iw ifl<i<s, and pa=sp ifi>s (3.279) 
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(since if all the servers are occupied, then the departure rate from the system 
is equal to su, for any number of persons in the system). 

Remark. M means that the waiting (and service) times are exponential, thus 
Markovian. | 


To end this subsection, we give the Kolmogorov backward and forward 
equations for a birth and death process. First, the backward equations are 


Po,j(t) = —Ao po,;(t) + Ao P1,5(t) (3.280) 
if i>0 
pi a(t) = pi pi-1g(t) — (Ai + pi) Dig (t) + Ai pi+i g(t) (3-281) 
In the case of the Kolmogorov forward equations, we find that 


Piot) = pai pi,i(t) — ro pio(t) (3.282) 
if 7>0 
piit) E Aj—1 pig—a(t) — (Aj + My) Dig (t) + Hy 41 Pigti(t) (3-283) 


3.3.5 Limiting probabilities and balance equations 


Since it is generally very difficult to solve explicitly the Kolmogorov equations 
to obtain the transition functions p,,;(t), we must often content ourselves with 
the computation of the limiting probability that the process will be in a given 
state when it is in equilibrium. To obtain these limiting probabilities, we can 
try to solve a system of linear equations called the balance equations of the 
process. 


Definition 3.3.12. Let To, 7, ... be nonnegative real numbers such that 
OO} 
> aS (3.284) 
j=0 


If the equation 


OQ 
X mipit) = 1; (3.285) 
i=0 


is satisfied for all j € {0,1,...} and for allt > 0, then n := (To, T1,...) isa 


stationary distribution. 


The expression stationary distribution is used because if we assume that 
P(X(0) =j] =r; for all j € {0,1,...} (3.286) 


then we have 


P[X(t) = j] = X PIX(t) = j | X(0) = P[X(0) = į 
i=0 
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= > pig (t) ri IS ny (3.287) 
7=0 
That is, 
PIX) = 9) =] P(X) =3] =a, (3.288) 


Vj € {0,1,...} and Vt>0. 


As in the discrete case, we can define the notion of recurrence of a state. 


Notation. We denote by T;,; the time elapsed between two consecutive visits 
to state i of the continuous-time Markov chain {X(t),t > O}. 


Definition 3.3.13. We say that state i is recurrent if P[T;,; < œ] =1 and 
transient if P[T; i < œ] < 1. Moreover, let 


ii EIT; | (3.289) 


The recurrent state i is said to be positive (respectively, null) recurrent if 
bee < œ (resp., = œ). 


We can prove the following theorem. 


Theorem 3.3.3. If the continuous-time Markov chain {X(t),t > 0} is ir- 
reducible and positive recurrent, then it has a unique stationary distribution 
T = (To, T1,- ..), where m; is the limiting probability 


Tj i= jim piy (t) for allj € {0, ee } (3.290) 


Remarks. i) Note that the 7;’s do not depend on the initial state 7. The 
quantity 7; is also the proportion of time that the Markov chain spends in 
state 7, on the long run. 


ii) If the Markov chain is transient or null recurrent, then it does not have a 
stationary distribution. 


iii) Contrary to the discrete-time Markov chains, a continuous-time Markov 
chain cannot be periodic. This follows from the fact that the time the pro- 
cess spends in an arbitrary state is a random variable having an exponential 
distribution, which is continuous. Consequently, if the limiting probabilities 
exist, then we say that the Markov chain is ergodic. 


iv) We find that 





Tj = for all j € Sx = {0,1,...} (3.291) 


Vihj j 
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where the state space Sx 4) may, of course, be finite, but must comprise more 
than one state. The interpretation of this result is the following: v; 1 is the 
average time that the process spends in state 7 on an arbitrary visit to this 
state, and uj; is the average time between two consecutive visits to state j, 
so that t/u;,; is the average number of visits to state j over a period of length 
t, where t is large. Thus, over this long period of time, the proportion of time 


during which the chain is in state j is effectively given by the ratio vy hy Mija 


v) If state j is absorbing, we consider it as positive recurrent. The theorem 
applies even in the case when the state space contains a single state, say 0. 
We then have, trivially, that 7 = 1. 


Now, if we differentiate both sides of Eq. (3.285) with respect to t, we 
obtain 


OO 
S mip s(t) =0 Vie {0,1,...} and Vt>0 (3.292) 
i=0 


With t = 0, we may write, by Proposition 3.3.9, that 


CO 
X minj =0 Yje {0,1,...} (3.293) 
i=0 


Remarks. i) If the number of states is finite, we can indeed interchange the 
derivative and the summation. However, when the state space is infinite, this 
interchange, though allowed here, must be justified. 

ii) It can be shown that Eq. (3.293) is satisfied if and only if Eq. (3.285) is 
also satisfied. 


Since Vij = —vj, we can rewrite Eq. (3.293) as follows: 
m=) miv; WIE {0,1,...} | (3.294) 
ifj 


Definition 3.3.14. The equations above are called the balance equations 
of the stochastic process {X (t), t > 0}. 


Remarks. i) We can interpret the balance equations as follows: the rate at 
which the process leaves state 7 must be equal to the rate at which it enters 
state 7, for all 7. Now, the rate at which transitions occur, on the long run, 
from state 7 to any other state 7 4 i of the Markov chain is given by 714,;, 
since 7; is the proportion of time that the process spends in state i during the 
period considered, and v;,; is the rate at which, when it is in state i, it enters 
j. Given that the process spends an exponential time with parameter v; in 
state J, 7;v; is the departure rate from j, and the sum of the terms Ti vij 
over all states 2 Æ j is the arrival rate to 7. 
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ii) The limiting probabilities 7; can be obtained by solving the system (3.294), 
under the condition $7797; = 1 [see (3.284)]. 

We will now obtain a general formula for the 7;’s when {X(t),t > O} isa 
birth and death process. First, we give a proposition that tells us when the 
T;’s do exist. 


Proposition 3.3.11. Let {X(t),t > 0} be an irreducible birth and death pro- 
cess, whose state space is the set {0,1,...}. We set 


 AgA + Àk- pli fia «++ Hk 
S = —— and So= saa 3.295 
i a HIH2'** Hk : Peaks l ) 


The Markov chain is 


positive recurrent if S1 < oo 
null recurrent if Sı = S2 = 00 (3.296) 
transient if Sy < œ% 


Remark. If the state space of the irreducible birth and death process is finite, 
then we necessarily have that Sı < œœ, so that the Markov chain is positive 
recurrent. 


With the help of this proposition, we can prove the following theorem. 
Theorem 3.3.4. For an irreducible and positive recurrent birth and death 


process {X(t),t > 0}, whose state space is the set {0,1,...}, the limiting 
probabilities are given by 


H; 
Tt =o forj =0,1,... (3.297) 
í aay fF j 
where 
Cees roe 
Ho:=1 and U1, := so aa forj 21 (3.298) 
HiH’ te Hj 


Proof. In the case of a birth and death process, the balance equations become 


state 7 departure rate from 7 = arrival rate to 7 


0 Agro = HITI 
1 (Ài + H1)mı = Hanz + Ant 
k (21) (Àk + Uk) Te = Hk41Tk41 + Àk-1Tk-1 


Adding the equations for 7 = 0 and 7 = 1, then for j = 1 and 7 = 2, etc., we 
find that 
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Nj; = [54174541 y J > 0 (3.299) 
which implies that 
a sasha e 
Tj = A m a A (3.300) 
So fag Bopa 


Using the fact that 77") m; = 1, we obtain 


—1 


AoA1** IIo 
Decree paH: eno Tk 
so that 
tm; = = a forj=0,1,... (3.302) 
Da Hy 


Finally, it can be shown that m = (To, 71,...) is a stationary distribution of 
the Markov chain {X (t), t > 0}. By Theorem 3.3.3, this distribution is unique. 
Thus, the limiting probabilities are indeed given by the preceding equations. 

0 


Remark. We can rewrite the formula (3.297) as follows: 


IL; , 
na f | ee ee 3.303 
Tj i45, Or j ( ) 


where Sı is defined in (3.295). 





Example 3.3.7. In the case of the two-state, continuous-time Markov chain 
(see pp. 129 and 137), we have 


À 
EE a 
Hı V4 


from which we deduce that 


1 Vi 1 i 
Ty = = = — 
OT IFE wn i 


V1 








Note that these results correspond to those given by Eq. (3.242). 
When vo = 4, we have that mo = mı = 1/2, which had to be the case, by 
symmetry. . 
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3.4 Exercises 


Section 3.2 


Question no. 1 

We suppose that the probability that a certain machine functions without 
failure today is equal to 
0.7 if the machine functioned without failure yesterday and the day before 
yesterday (state 0), 
0.5 if the machine functioned without failure yesterday, but not the day before 
yesterday (state 1), 
0.4 if the machine functioned without failure the day before yesterday, but 
not yesterday (state 2), 
0.2 if the machine did not function without failure, neither yesterday nor the 
day before yesterday (state 3). 


(a) Find the one-step transition probability matrix of the Markov chain asso- 
ciated with the functioning state of the machine. 


(b) Calculate py. that is, the probability of moving from state 0 to state 1 
in two steps. 


(c) Calculate the average number of days without failure of the machine over 
the next two days, given that the Markov chain is presently in state 0. 


Question no. 2 
Let {Xn n = 0,1,...} be a Markov chain whose state space is the set 
{0,1} and whose one-step transition probability matrix P is given by 


p- F A 


where 0< p< 1. 
(a) Suppose that p = 1 and that Xp = 0. Calculate E[X9]. 
(b) Suppose that p = 1/2 and that P[Xo = 0] = P[Xo = 1] = 1/2. We define 
the continuous-time stochastic process {Y (t), t > 0} by Y(t) = ¢Xiy, for t > 0, 
where |t] denotes the integer part of t. 
(i) Calculate Cy (t,t + 1). 
(ii) Is the stochastic process {Y (t), t > 0} wide-sense stationary? Justify. 
(iii) Calculate lim, 0 P[X, = 0). 


Question no. 3 

We consider a Markov chain {X,,n = 0,1,...} having states 0 and 1. 
On each step, the process moves from state 0 to state 1 with probability p € 
(0,1), or from state 1 to state 0 with probability 1 — p. 
(a) Calculate py A 
(b) Suppose that Xo = 0. Calculate the autocorrelation function Rx (1, 13). 
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Question no. 4 
The one-step transition probability matrix P of a Markov chain whose 
state space is {0,1} is given by 


p= ka a 


Calculate E[X2] if P[Xo = 0] = 1/3. 
Question no. 5 

Let Y1, Y2,... be an infinite sequence of independent random variables, 
all having a Bernoulli distribution with parameter p = 1/3. We define Xn = 
Yi: Yk, for n = 1,2,.... Then (see p. 85) the stochastic process {Xn, n = 


1,2,...} is a Markov chain. Calculate Dh >- 


Question no. 6 
Let {Xn n = 0,1,...} be a random walk for which 


2 1 
Pi i+ = 3 and Pi i—1 = 3 for i € {0, al ies oy ee af 
Calculate E[X2 | Xo = 0]. 


Question no. 7 
Let {X,,n = 0,1,...} be a Markov chain whose state space is the set 
{0,1} and whose one-step transition probability matrix is given by 


01 
P= [10 
(a) Calculate Cx (t1, ta) at ti = 0 and ta = 1 if P|Xo = 0] = P[Xo = 1] = 
1/2: 
(b) Find limno P[X, = 0 | Xo = 0]. 


Question no. 8 
Let X1, X2,... be an infinite sequence of independent random variables, 
all having a Poisson distribution with parameter a = 1. We define 


n 
Ye A for n = 1,2,... 
k=1 


Then, {Yn, n = 1,2,...} is a Markov chain (see p. 85). Calculate po that is, 
the probability of moving from state 1 to state 3 in four steps. 


Question no. 9 
The flow of a certain river can be one of the following three states: 


0: low flow 
1: average flow 
2: high flow 
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We suppose that the stochastic process {X,,n = 0,1,...}, where Xn repre- 
sents the state of the river flow on the nth day, is a Markov chain. Furthermore, 
we estimate that the probability that the flow moves from state 7 to state j 
in one day is given by the formula 


1 ae 
Pij = 5 jt — j|; 
where 0 < 6; < 1, for i,j = 0,1,2. 


(a) Calculate the probability that the river flow moves from state 0 to state 
1 in one day. 


(b) What is the probability that the river flow moves from state 0 to state 2 
in two days? 


Question no. 10 

A machine is made up of two components that operate independently. 
The lifetime T; (in days) of component i has an exponential distribution with 
parameter A;, for i = 1, 2. 

Suppose that the two components are placed in parallel and that Ay = 
Ag = In2. When the machine breaks down, the two components are replaced 
by new ones at the beginning of the following day. Let Xn be the number 
of components that operate at the end of n days. Then the stochastic pro- 
cess {Xn, n = 0,1,...} is a Markov chain. Calculate its one-step transition 
probability matrix. 


Question no. 11 
Let {X,,n = 0,1,...} be a Markov chain whose state space is the set 
{0,1,2,3,4} and whose one-step transition probability matrix is 


1 0 000 
0.5 0.2 0.3 0 0 
P=] 0 0 010 
0 0 00i 
0 0 1 00 


(a) Calculate the probability that the process will move from state 1 to state 
2 in four steps. 


(b) Suppose that Xo = 1. Let N; be the number of times that state 1 will be 
visited, including the initial state. Calculate ÆN]. 


Question no. 12 
A Markov chain {X,,n = 0,1,...} with state space {0,1,2,3} has the 
following one-step transition probability matrix: 


1/21/2 0 0 
1/21/41/4 0 
0 0 1/43/4 
000 1 


P= 
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Assuming that Xo = 1, calculate the probability that state 0 will be visited 
before state 3. 


Question no. 13 
A Markov chain has the following one-step transition probabilities: 


Poo = 1 
Pa=p=l—py-1 fort =1,2,3,.... 


Calculate the probability roe that the chain will move from state i to state 0 
for the first time after exactly n transitions, for i = 1,2,... 


Question no. 14 


Let 
qp000 


gO0p00 
P= |O0q0p0 

O0g0p 

O00gp 
where p+q=1 and 0 < p < 1, be the one-step transition probability matrix 
of a Markov chain whose state space is the set {0,1, 2,3, 4}. 
(a) Is the chain periodic or aperiodic? Justify. 
(b) Calculate, if they exist, the limiting probabilities 7;. 
Question no. 15 


We perform repeated trials that are not Bernoulli trials (see p. 51). We 
suppose that the probability pn of a success on the nth trial is given by 


1/2 forn=1,2 
Pn = Xn tl 
Xn +2 





rn s= 3A 


where Xn is the total number of successes obtained on the (n — 2)nd and 
(n — 1)st trials. Calculate liMmp—oo Pn. 


Question no. 16 

In the gambler’s ruin problem (see p. 101), let Y; be the number of plays 
needed to end the game (with the player being ruined or having reached his 
objective of k units), given that his initial fortune is equal to 7 units, for 
i = 0,1,... ,k. Show that 


Opsi [e eg - i if p # 1/2 


E[Y;] = 
i(k — i) if p= 1/2 


Indication. We may write that E[Yo] = E[Y;] = 0 and (by conditioning on 
the result of the first play) 
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ElY;|] = 1 +p E|Yi+1] +40 E[Y;-ı] fori=1,...,k-1 


Question no. 17 

We consider a branching process for which p; = (1 — p)'p, for i = 0,1,..., 
where 0 < p < 1. That is, Y := Z +1, where Z is the number of descendants of 
an arbitrary individual, has a geometric distribution with parameter p. Show 
that the probability qo of eventual extinction of the population is given by 


wz p/(l—p)ifp < 1/2 
j 1 ifp>1/2 


Question no. 18 
Let {Xn, n = 0,1,...} be an irreducible and ergodic Markov chain. Sup- 
pose that the chain is in equilibrium, so that we can write that 


P|Xn =i] =m; for all states i and for any n 


We can show that the process {X,,k =...,n+1,n,...}, for which the time 
is reversed, is also a Markov chain, whose transition probabilities are given by 


Ti _ 
i,j i= Pike So xy a = Pii — for all states 7,7 
i 
We say that the chain {X,,n = 0,1,...} is time-reversible if qij = Pij, for 
all 7,7. Show that the Markov chain whose state space is the set {0, 1,2, 3} 
and with one-step transition probability matrix 


1/21/20: 0 
1/2 0 1/2 0 
0 12 0 1/2 
0 0 1/21/2 


P= 


is time-reversible. 


Question no. 19 

In the urn model of P. and T. Ehrenfest for the movement of molecules 
in a gas (see p. 75), suppose that Xp = 1. Let mn := E[X,]| be the average 
number of molecules in urn I after n shifts. Show that 


m aes tee ‘ pa 
oo N 2 


Indication. Show first that mn, = 1 + cae Mtr 


for m= i 2na 


Question no. 20 
A Markov chain has the following one-step transition probabilities: po,ı = 1 
and 
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Diiti =a, (>0)=1—-p,o fori =1,2,3,... 


(a) Show that all states are recurrent if and only if 


Tt 
lim I] a, =0 
Te CO 
k=1 


(b) Show that if the chain is recurrent, then all states are positive recurrent 


if and only if 
OO n 
S [Io < o 
n=1Lk=1 


Indication. If N is a random variable taking its values in the set N° := 
{0,1,...}, then we have 


E[N] = 3 PIN > n] 


n=i 


Use this result with the variable N denoting the number of transitions needed 
to return to state 0. 


Question no. 21 
The one-step transition probabilities of a Markov chain whose state space 
is {—3, —2, —1, 0, 1,2,3} are given by 


Pii+1 = 2/3 fori = —3,-2,...,2 
Pai =1/3 fori = —2,-1,...,3 
p-3,3 = 1/3 = 1 — p3,-3 


Remark. This chain can be considered as a particular case of a random walk 
defined on a circle. 


(a) Show that the Markov chain is irreducible. 
(b) Determine the period of the chain. 


(c) Calculate the fraction of time that the process spends, over a long period, 
in state i, for i = —3,... ,3. 


Question no. 22 

(a) Show that in a symmetric random walk {Xn, n = 0,1,...}, starting from 
Xo = 0, the probability that state a > 0 will be visited before state —b < 0 is 
equal to b/(a + b). 

(b) In a random walk starting from Xo = 0, what is the probability that there 
will be exactly one visit to state 0? 

(c) In a random walk on {0,1,2,...}, starting from Xo = 0 and for which 
Po,1 = 1, what is the probability that there will be exactly k € {1,2...} 
visit(s) to state 0? 
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Question no. 23 
We consider a branching process {Xn, n = 0,1,...} for which Xo = 1 and 
x 
p= i fori =U Fy. 
i! 
That is, the number of descendants of an arbitrary individual has a Pois- 
son distribution with parameter A. Determine the probability go of eventual 
extinction of the population if (a) A = In2 and (b) A = In4. 


Question no. 24 
A Markov chain with state space {-2,—1,0,1,2} has the following one- 
step transition probabilities: 


Pii+1 = 3/4 fori = —2,-1,0,1 
Dii = l4 Fort 1,042 
p-22 = 1/4 = 1 — p2,-2 


(a) Determine the classes of the Markov chain. For each class, establish 
whether it is recurrent or transient and whether it is periodic or aperiodic. 


(b) Do the limiting probabilities exist? If they do, compute them. 
(c) Is the chain time-reversible (see p. 147)? Justify. 


Question no. 25 
We consider an irreducible Markov chain whose state space is the set 
{0, 1,2,3,4} and whose one-step transition probability matrix is given by 


01/32/3 0 0 
00 0 1/43/4 
P=|00 0 1/43/4 
10 0 0 0 
10 0 0 0 


(a) What is the period of the chain? . 


(b) What is the fraction of time, 7;, that the process spends in state j on the 
long run, for 7 = 0,1, 2,3, 4? 


Question no. 26 

Let (Xn, Yn) be the position of a particle that moves in the plane. We 
suppose that {X,,n = 0,1,...} and {Y,,n = 0,1,...} are two independent 
symmetric random walks such that Xo = Yo = 0. We define Dn = Y X2 + Y/?. 
That is, D, represents the distance of the particle from the origin after n 
transitions. Show that if n is large, then 


P|Da <d x1- e” for d>0 


Indication. Express X,, (and Yn) in terms of a binomial distribution, and use 
the fact that if Zi, Z2,... ,Zę are independent random variables having a 
standard Gaussian distribution, then 
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Zi FZ +t Zin Gl(a=k/2,A\ =1/2) 


Question no. 27 

Let Y1, Y2,... be an infinite sequence of independent random variables, 
all distributed as the discrete r.v. Y whose set of possible values is the set 
Z := {0,+1,+2,...} of all integers. We know (see p. 85) that the stochastic 
process {Xn, n = 1,2,...} defined by 


Xn =¥, +Y +... + Yn for == 1 2,5: 


is a Markov chain and that pij = aj-;, where a; := P[Y = i], for any 1. 
We suppose that |E[Y]| < oo and V[Y] < oo. Show that the chain is 
transient if E[Y] 4 0. 


Question no. 28 
We consider a Markov chain with transition probabilities given by 


and Pkk+1 = [5 


O k+i 
Pko = 79 k42 


fOr K=O Ae 2st 

(a) Show that the chain is irreducible and ergodic. 
Indication. See Question no. 20. 

(b) Calculate the limiting probabilities mg, for k = 0,1,2,.... 


Question no. 29 
The one-step transition probabilities of an irreducible Markov chain are 
given by 
1 k 
po =l, Pko = Ral’ and = Pk,k+1 = ra 
LOR A ees 
(a) Calculate Doe: to) a as Bay een 


(b) Calculate py. for k = 2 and 3. What is the period of the chain? 

(c) Is the Markov chain transient, positive recurrent, or null recurrent? Justify. 
Indication. See Question no. 20. 

Question no. 30 


We consider a Markov chain defined by the one-step transition probability 
matrix 
1/2 1/4 1/4 
P=]/]ati-a 0 
0 a l~a 
where 0 < a < 1. The state space is the set {0, 1,2}. 


(a) For what values of the constant a is the Markov chain irreducible and 
ergodic? Justify. 
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(b) Calculate the limiting probabilities for the values of a found in part (a). 


Question no. 31 

A system is made up of two identical components in standby redundancy. 
That is, only one component is active at a time, and the other is in standby. 
We assume that the lifetime of each component has an exponential distri- 
bution and that the probability that the active component will fail during a 
given day is equal to 0.1. The other component, if it is not also down, then 
relieves the failed one at the beginning of the following day. There is a single 
technician who repairs the components. Moreover, he only starts repairing a 
failed component at the beginning of his workday, and he needs two working 
days to complete a repair. 

Let Xn be the condition of the components at the end of the nth day. The 
process {Xn, n = 0,1,2,...} is a Markov chain having the following states: 


0: neither component is down 

1: a single component is down and it will be repaired one day 
from now 

2: a single component is down and it will be repaired two days 
from now 

3: both components are down (and a component will be repaired 
one day from now) 


(a) What is the matrix P of one-step transition probabilities of the chain? 


(b) For each class of the chain, determine whether it is transient or recurrent. 
Justify. 


(c) Suppose that Xp = 0. Calculate the probability that there will be a single 
component that is not down at the end of the second day. 


Question no. 32 


Let 
1/3 01/3 1/3 
100 0 
P= 010 0 
010 0 


be the one-step transition probability matrix of an irreducible Markov chain 
whose state space is the set {0,1,2,3}. 


(a) What is the period of state 1? Justify. 


(b) Calculate, if they exist, the quantities 7;, where 7; is the proportion 
of time that the process spends in state j, over a long period of time, for 
a= 0.1.2.3. 

(c) Suppose that the elements of the first row of the matrix P are the p;’s 
of a branching process. Calculate the probability that the population will die 
out if (i) Xo = pA (ii) Xı <1 (and Xo = 2j: 
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Question no. 33 

John plays independent repetitions of the following game: he tosses two 
fair dice simultaneously. If he gets a sum of 7 or 11 (respectively, 2, 3, or 12), 
he wins (resp., loses) $1. Otherwise, he neither wins nor loses anything. John 
has an initial fortune of $i, where i = 1 or 2, and he will stop playing when 
either he goes broke or his fortune reaches $3. Let X, be John’s fortune after 
n repetitions, for n = 0,1,.... Then {X,,n = 0,1,...} is a Markov chain. 


(a) Find the matrix P of one-step transition probabilities of the Markov chain. 


(b) For each class of the chain, specify whether it is recurrent or transient. 
Justify. 

(c) Calculate the mathematical expectation of X, if the initial distribution of 
the chain is ag = 0, ay = 1/2, ag = 1/2, and a3 = 0. 


Question no. 34 
A Markov chain whose state space is the set {0,1,2} has the owe 
one-step transition probability matrix: 


0 10 
P= |{1-p0p 
0 10 


where 0 < p< 1. 
(a) Calculate P™), for n > 2. 
(b) Find the period of every state of the Markov chain. 
(c) (i) Calculate the proportion of time that the process spends, on the long 
run, in state 0. 
(ii) Is this proportion equal to the limit limp—oo pie o? )2 J ustify. 


Question no. 35 

A machine is made up of two components placed in parallel and that oper- 
ate independently of each other. The lifetime (in months) of each component 
has an exponential distribution with mean equal to two months. When the 
machine breaks down, the two components are replaced at the beginning of the 
next month. Let Xn, n = 0,1,..., be the number of components functioning 
after n months. 


(a) Justify why the stochastic process {X,,n = 0,1,...} is a Markov chain. 


(b) Calculate the matrix P of one-step transition probabilities of the Markov 
chain. 


(c) Identify each class of the chain as recurrent or transient. Justify. 
(d) Calculate V[X,] if the two components are functioning at time n = 0. 
Question no. 36 


A Markov chain whose state space is the set {0,1,...} has the following 
one-step transition probabilities: 
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Pio =2/3 and p;p = (1/4) fori=0,1,... andk=1,2,... 


(a) Show that the limiting probabilities 7; exist, and calculate them. 
(b) Suppose that the p;,’s are the p,’s, k > 0, of a branching process. Calcu- 
late the probability of eventual extinction of the population if Xo = 1. 


Question no. 37 
Let X1, X2,... be an infinite sequence of independent random variables, 
all having a Bernoulli distribution with parameter p € (0,1). We define 


n 
= So Xp FOP Fs 
k=1 
Then the stochastic process {Y,,,n = 1,2,...} is a Markov chain (see p. 85). 


(a) Calculate the one-step transition probability matrix P of the Markov 
chain. 


(b) For every class of the chain, find out whether it is recurrent or transient. 
Justify. 

(c) Calculate V[Y, | X4 = ll. 

(d) Let Tı := min{n > 1: Yp = 1}. What is the distribution of the random 
variable T;? 


Question no. 38 


Let 
0 1 0 0 
P 1/2 0 1/2 0 
EID? -0172 
0 0 1 0 


be the one-step transition probability matrix of a Markov chain {X,,n = 
0,1,...} whose state space is the set {0,1, 2,3}. 

(a) Let 7; be the proportion of time that the process spends in state j, over 
a long period of time. Show that the z,’s exist and calculate them. 

(b) Calculate the period of the Markov chain. 

(c) Let Tj := min{n > 0: Xn = j}. Calculate P[Zo < Ts | Xo = 1]. That 
is, calculate the probability that, from state 1, the process will visit state 0 
before state 3. 


Question no. 39 
We define the Markov chain {Y,,n = 1,2,...} by 


na 
Yes Y Xa for n = 1,2,... 
k=1 


where X1, X2,... is an infinite sequence of independent random variables 
having a binomial distribution with parameters n = 2 and p = 1/2 (see p. 85). 
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(a) Calculate the transition matrix of the Markov chain. 


(b) For every class of the chain, establish whether it is recurrent or transient. 
Justify. 
(c) What is the distribution of the random variable Y2? 
(d) Let T; := min{n > 1: Yp = 1}. Calculate 
(i) P[T, =k]; fork =1,2,..., 
(ii) P(T, < oo]. 


Question no. 40 
We consider the particular case of the gambler’s ruin problem (see p. 101) 
for which k = 4. 


(a) (i) Calculate the matrix P if the gambler, on any play, actually has a 
probability equal to 


1/4 of winning the play 
1/4 of losing the play 
1/2 of neither winning nor losing the play 


(ii) Do the probabilities 7} := limp—oo PIXn = j | Xo = 2] exist? If they 


do, calculate these probabilities. 


(b) Suppose that the gambler, on an arbitrary play, bets 


$1 if his fortune is equal to $1 or $3 
$2 if his fortune is equal to $2 


Calculate, under the same assumptions as in (a), the probability that the 
gambler will eventually be ruined if Xp = $3. 


Question no. 41 

We consider a population of constant size, N, composed of individuals of 
type A and of type B. We assume that before reproducing, and disappearing 
from the population, an arbitrary individual of type A (respectively, B) is 
mutated into an individual of type B (resp., A) with probability a (resp., 8). 
Moreover, we assume that at the moment of reproduction, each individual in 
the population has a probability p of giving birth to an offspring of type A, 
where p is the mathematical expectation of the proportion of individuals of 
type A in the population after mutation. Finally, we assume that the individ- 
uals are independent from one another. Let X,, be the number of individuals 
of type A in the nth generation before mutation. Then {X,,n = 0,1,...} is 
a Markov chain. 


(a) Calculate p (= p;) if Xn = å. 

(b) Calculate p; j, for i,j € {0,..., N}. 

(c) Suppose that a = 0 and 8 € (0,1). For each class of the Markov chain, 
determine whether it is recurrent or transient. Justify. 
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(d) Suppose that there are exactly N4 > 0 individuals of type A in the initial 
generation after mutation. Given that the first of these N4 individuals gave 
birth to an offspring of type A, calculate, with p = N4/N, (i) the mean and 
(ii) the variance of X1. 


Question no. 42 
Suppose that 


1/4.1/2 1/4 
P=] &a 01-a 
B 01-8 


is the one-step transition probability matrix of a Markov chain with state 
space {0, 1,2}. 

(a) For what values of a and 8 is the Markov chain irreducible? 

(b) Suppose that a = 8 = 1/2. Compute, if they exist, the limiting probabil- 
ities 7;, for 7 = 0,1, 2. 

(c) Suppose that Xo = 0. Compute the probability that the process will (i) 


visit state 1 before state 2 and (ii) return to (or stay in) state 0 before visiting 
state 2. 


Question no. 43 
Let {Xn, n =0,1,...} be a Markov chain with state space {0,2} and with 
one-step transition probability matrix 


ex- [a] 


and let {Y,,n = 0,1,... } be a Markov chain whose state space is {3,4} and 
whose matrix P is given by 
01 
Py = [i0] 


Assume that the random variables X, and Y, are independent, for all n. 
We define Zn = Xn + Yn. We can show that the stochastic process {Zn, n = 
0,1,...} is a Markov chain whose state space is {3, 4, 5, 6}. 

(a) Calculate the matrix Pz of the Markov chain {Z,,n = 0,1,... }. 
(b) For every class of the chain {Z,,n = 0,1,...}, determine whether it is 
recurrent or transient. Justify. 


(c) Give the period of every class of the chain {Z,,n = 0,1,... }. 


(d) Calculate the matrix py and deduce from it the value of the limit lim, —o 
P\Z47=3:;|Zo=31. 


Question no. 44 
Let 
0 1/2 1/2 
P=|0 a i-a 
Bao 20 
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be the one-step transition probability matrix of a Markov chain {X,,n = 
1,...} having state space {0, 1,2}. 

(a) For what values of a and 8 is the Markov chain irreducible? 

(b) Suppose that a = 1/2 and 8 = 1/3. Calculate, if they exist, the limiting 

probabilities 7;, for 7 = 0,1, 2. | 

(c) Give the values of a and 8 for which the limiting probabilities 7; exist 

and are equal to 1/3, for j = 0,1, 2. Justify. 


(d) Suppose that a = 1/2, 8 = 1, and Xo = 1. Let No be the number of 
transitions needed for the process to visit state 0 for the first time. Calculate 
E| No]. 


Question no. 45 

A person buys stocks of a certain company at the price of 3 cents per share 
(what is known as a penny stock). The investor decides to sell her shares if 
their value decreases to 1 cent or becomes greater than or equal to 5 cents. 
Let Xn be the value of the shares (for the investor) after n days. We sup- 
pose that {X,,n = 0,1,...} is a Markov chain having as state space the set 
{0,1,... , 7}, and for which rows 3 to 5 (corresponding to states 2,3, 4) of the 
one-step transition probability matrix P are the following: 


1/8 1/41/41/4 1/8 0 0 0 
1/12 1/7 1/41/21 1/4 1/71/12 0 
O 1/121/7 1/4 1/21 1/4 1/7 1/12 


(a) Give the other rows of the matrix P. 

Indication. Once the investor has sold her shares, or their price went to zero, 
their value (for the investor) does not change anymore. 

(b) For every class of the chain {Z,,n = 0,1,...}, establish whether it is 
recurrent or transient. Justify. 

(c) Calculate E[X, | {X1 = 3} U {X; = 4}. 

(d) What is the probability that, after exactly two days, the investor 50) sells 


his shares with a profit? (ii) sells her shares with a loss? (iii) loses all the 
money she invested? 


Question no. 46 
A Markov chain {X,,n = 0,1,...} with state space {0,1,2} has the fol- 
lowing one-step transition probability matrix: 


1/20 1/2 


P=) 01 0 
1 0 0 


Calculate liMmn—=oo P; ee for all i, 7 € {0,1,2}. 
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Question no. 47 

We consider the gambler’s ruin problem (see p. 101). Suppose that Xo = 
$1 and that p = 1/4. However, if the player wins, he wins $2 and if he loses, 
he loses $1. His objective is to reach at least $4. Calculate the probability that 
he will achieve his objective. 


Question no. 48 
(a) Calculate the probability gg of eventual extinction of the population in a 
branching process for which pp = 1/4, pı = 1/4, and pg = 1/2. 


(b) Suppose that the individuals can only give birth to twins, so that the 
probabilities in (a) become pj = 1/4, pš = 1/4, and pă = 1/2. Can we assert 
that qù = q2? Justify. 


Question no. 49 
Let {X,,n = 0,1,...} be a symmetric random walk defined on the set 
{0,+1,+2,...} and such that Xp = 0. We set 


Yn = X? for n = 0,1,... 


It can be shown that {Y,,n = 0,1,...} is a Markov chain whose state space 
is {0,1,4,9,...}. 

(a) Calculate the one-step transition probability matrix of the chain {Y,,n = 
0,1,...}. 


(b) Is the stochastic process {Y,,n = 0,1,...} a random walk? Justify. 


(c) For each class of the chain {Y,,n = 0,1,...}, (i) determine whether it is 
transient or recurrent and (ii) find its period. 


Question no. 50 
Let 
1/201/41/4 
010 0 
a 100 0 
0 01/43/4 


be the one-step transition probability matrix of a Markov chain {Xn,n = 


0,1,...} whose state space is {0,1, 2,3}. Calculate the limit limp oo ee for 
all i, j € {0, 1,2, 3}. 


Question no. 51 

Suppose that Xo = $i in the gambler’s ruin problem (see p. 101) and 
that p Æ 1/2. Suppose also that if the player loses, then someone lends him 
(only once) $1 and he starts to play again, independently from what occurred 
previously. However, the probability p becomes p/2. His objective is to reach 
$k (without taking into account the dollar that someone may have lent him), 
where k > i. Calculate the probability that he will achieve his objective. 
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Question no. 52 
(a) Calculate, assuming that Xo = 1, the probability qo of eventual extinction 
of the population in a branching process for which p; = 1/4, for i = 0,1, 2, 3. 


(b) Find another distribution of the p;’s for which the value of the probability 
qo is the same as that in (a). 


Question no. 53 

A machine is composed of two components. The lifetime T; of component 
i has an exponential distribution with parameter 4,;, for 7 = 1,2. When the 
machine breaks down, a technician replaces the failed component(s) at the 
beginning of the next time unit. Let Xn, n = 0,1,..., be the number of 
components that are not down after n time unit(s). 


(a) Is the stochastic process {Xn, n = 0,1,...} a Markov chain if the compo- 
nents are placed in series? If it is, justify and calculate the one-step transition 
probability matrix P of the chain. If it’s not, justify. 


(b) Suppose that the components are placed in parallel and operate indepen- 
dently from each other, but that only one component is active at a time. That 
is, the components are in standby redundancy. In this case, {Xy,n = 0,1,...} 
is a Markov chain. Calculate its transition matrix if A, = A2. 


Question no. 54 

Suppose that, in the preceding question, the components are placed in 
parallel and operate (independently from each other) both at the same time. 
We say that they are in active redundancy. 
(a) The stochastic process {X,,n = 0,1,...} ¿s a Markov chain. Calculate its 
matrix P. 


(b) Let 
0 if neither component is operating 
1, if only component no. 1 is operating 
A = A i i 
l> if only component no. 2 is operating 
2 if both components are operating 
after n time unit(s), for n = 0,1,.... Is the stochastic process {Y,,n = 


0,1,...} a Markov chain? If it is, justify and calculate the transition matrix 
of the chain. If it’s not, justify. 


Question no. 55 

A system comprises two components placed in parallel and operating (both 
at the same time) independently from each other. Component 7 has an expo- 
nential lifetime with parameter A;, for i = 1,2. Let Xn, n = 0,1,..., be the 
number of active components after n time unit(s). 


(a) Suppose that, after each time unit, we replace the failed component(s). 
Calculate the one-step transition probability matrix P of the Markov chain 
{Xp,n =0,1,...}. 

(b) It can be shown that the limiting probabilities 7; exist in part (a). Cal- 
culate these limiting probabilities. 
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(c) Suppose that A; = Ag and that, after each time unit, we replace a single 
component, only when the system is down. Moreover, assume that Xo = 2. 
(i) Calculate the matrix P of the Markov chain {X,,n = 0,1,... }. 
(ii) What is the number of classes of the chain? 


Question no. 56 

We consider a symmetric random walk in two dimensions (see p. 75), whose 
state space is the set {(i, j): i = 0, 1,2; j = 0,1, 2}. Moreover, we suppose that 
the boundaries are reflecting. That is, when the process makes a transition 
that would take it outside the region defined by the state space, then it returns 
to the last position it occupied (on the boundary). 


(a) Calculate the one-step transition probability matrix of the Markov chain. 


(b) Show that the limiting probabilities exist and calculate them. 


Question no. 57 

A player has only one money unit and wishes to increase his fortune to five 
units. To do so, he plays independent repetitions of a game that, in case of a 
win, yields double the sum he betted. In case of a loss, he loses his bet. On 
each play, he bets an amount that, if he wins, enables him either to exactly 
reach his objective or to get as close as possible to it (for example, if he has 
three units, then he bets only one). We suppose that the game ends when the 
player either has achieved his target or has been ruined and that he has a 
probability equal to 1/2 of winning an arbitrary play. 


(a) What is the probability that he reaches his target? 


(b) What is the average number of repetitions needed for the game to end? 


Question no. 58 

A particle moves in the plane according to a two-dimensional symmetric 
random walk (see p. 75). That is, the particle has a probability equal to 
1/4 of moving from its current position, (Xn, Yn), to any of its four nearest 
neighbors. We suppose that the particle is at the origin at time n = 0, so that 
Xo = Yo = 0. Thus, at time n = 1, the particle will be in one of the following 
states: (0,1), (0,—1), (1,0), or (—1,0). Let 


DŽ := X? + Y? 
be the square of the distance of the particle from the origin at time n. Calculate 
E[D}]. 


Question no. 59 

Show that for a symmetric random walk of dimension k = 2 (see Question 
no. 58), the probability that the number of visits to already visited states will 
be infinite is equal to 1. Generalize this result to the case when k € N. 
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Question no. 60 

A machine is composed of two identical components placed in series. The 
lifetime of a component is a random variable having an exponential distribu- 
tion with parameter u. We have at our disposal a stock of n — 2 new compo- 
nents that we differentiate by numbering them from 3 to n (the components 
already installed bearing the numbers 1 and 2). When the machine fails, we 
immediately replace the component that caused the failure by the new com- 
ponent bearing the smallest number among those in stock. Let T be the total 
lifetime of the machine, and let N be the number of the only component that, 
at time T, will not be down. Find (a) the probability mass function of N, (b) 
the mathematical expectation of T, and (c) the distribution of T. 


Question no. 61 

We use k light bulbs to light an outside rink. The person responsible for the 
lighting of the rink does not keep spare light bulbs. Rather, he orders, at the 
beginning of each week, new light bulbs to replace the ones that burned out 
during the preceding week. These light bulbs are delivered the following week. 
Let X,, be the number of light bulbs in operation at the beginning of the nth 
week, and let Y„ be the number of light bulbs that will burn out during this 
nth week, for n = 0,1,... . We assume that, given that X, = i, the random 
variable Y, has a discrete uniform distribution over the set {0,1,... , a}: 


PYa =j | Xa =i] = 25 forj =0, lenn and E0 k 


(a) Calculate the one-step transition probability matrix of the Markov chain 
Aers (ee eran 

(b) Show that the limiting probabilities of the chain {X,,, = 0,1,...} exist 
and are given by 


2(i+1) 


SOON cd ad oe ee E, 
(k+1)(k+2) ~- 


Ni 


Question no. 62 

Electric impulses are measured by a counter that only indicates the highest 
voltage it has registered up to the present time instant. We assume that the 
electric impulses are uniformly distributed over the set {1,2,... , N}. 


(a) Let Xn be the voltage indicated after n electric impulses. The stochastic 
process {X,,n = 1,2,...} is a Markov chain. Find its one-step transition 
probability matrix. 


(b) Let m; be the average number of additional impulses needed for the 
counter to register the maximum voltage, N, when the voltage indicated is 2, 
foris Ll Zea N As 

(i) Obtain a set of difference equations (as in the absorption problems, 
p. 100) for the m,’s, and solve these equations to determine m;, for all i. 
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(ii) Calculate directly the value of the m,’s without making use of the 
difference equations. 


Question no. 63 

Let X1, Xo, ... be an infinite sequence of independent and identically 
distributed random variables, such that px, (x) = 1/3 if x = —1,0,1. The 
stochastic process {Y,,7 = 1,2,...} defined by 


neS DEn E l lys 


i=1 


is a Markov chain (see p. 85). 


(a) Calculate the one-step transition probability matrix P of the chain. 


(b) Give an exact formula for Dos orke pee 


h 


, I , k 
(c) Use the central limit theorem to obtain an approximate formula for Bo 


when k is large enough. 


Question no. 64 

A player has $900 at his disposal. His objective is reach the amount of 
$1000. To do so, he plays repetitions of a game for which the probability 
that he wins an arbitrary repetition is equal to 9/19, independently from one 
repetition to another. 


(a) Calculate the probability that the player will reach his target if he bets 
$1 per repetition of the game. 


(b) Calculate the probability that the player will reach his target if he adopts 
the following strategy: he bets 


$(1000 — x) if 500 < x < 1000 
$z if 0 < x < 500 


where x is the amount of money at his disposal at the moment of betting. 


(c) What is the expected gain with the strategy used in (b)? 


Question no. 65 

We consider a system made up of two components placed in parallel and 
operating independently. The lifetime Tı of component no. 1 has an expo- 
nential distribution with parameter 1, while that of component no. 2 is a 
random variable Tə ~ Exp(1/2). When the system fails, 50% of the time the 
two components are replaced by new ones, and 50% of the time only the first 
(of the two components) that failed is replaced. Let Xn = 1, (respectively, 
l2) if only component no. 1 (resp., no. 2) is replaced at the moment of the 
nth failure, and X,, = 2 if both components are replaced. We can show that 
{Xn n =1,2,...} is a Markov chain. 


(a) Find the one-step transition probability matrix of the chain. 


162 3 Markov Chains 


(b) For each class of the chain, specify whether it is transient or recurrent. 
Justify. 


(c) Calculate ps, for BSc 2 exes 
(d) What is the period of state 12? Justify. 


Question no. 66 


In the gambler’s ruin problem (see p. 101), suppose that the player has an 
infinite initial fortune and that p = 1/2. Let X, be the gain (or the loss) of 
the player after n repetitions of the game. Suppose also that when Xn < 0, 
the player plays double or quits. That is, if Xn = —1, then Xn+ı = 0 or —2; 
if Xn = —2, then Xj41 = 0 or —4, etc. The process {Xn,n = 0,1,...} isa 
Markov chain whose state space is the set {... ,—4, —2, —1,0,1,2,... }. 


(a) Find the one-step transition probability matrix of the chain. 

(b) Suppose that the state space is {—4, —2, —1,0, 1,2} and that p-4,-4 = 
p22 = 1/2 [the other probabilities being as in (a)]. Calculate, if they exist, 
the limiting probabilities. | 

(c) Suppose now that the player decides to stop playing if his losses or his 
profits reach four units. Calculate the probability that the player will stop 
playing because his losses have reached four units, given that he won the first 
repetition of the game. 


Question no. 67 
The state space of a Markov chain {X,,n = 0,1,2...} is the set of non- 
negative integers {0,1,2,...}, and its one-step transition probability matrix 
is given by 
O 1 0 
1/2 0 1/2 0 
P = | 1/3 1/3 0 1/3... 

1/41/41/4 0 1/40... 

(a) Calculate i. 


(b) For each class of the chain, determine whether it is transient or recurrent. 
Justify. 

(c) Find the period of each class. 

(d) Let T;,; be the number of transitions needed for the process to move from 
state i to state j. Calculate PITi o < Ti, | {Ti,0 < 2} U{Ti3 < 2}]. 


Question no. 68 
We consider a Markov chain whose state space is the set {0,1,2,...} and 
for which 
Pij =p; >0 VWi,7 € {0,1,2,...} 


Calculate, assuming they exist, the limiting probabilities of the chain. 
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Question no. 69 


Suppose that, in the gambler’s ruin problem (see p. 101), Xo = 1 and 
k = 4. Moreover, suppose that the value of p is not constant, but rather 
increases with X,. More precisely, the one-step transition probability matrix 
of the chain {X,,n =0,1,2...} is 


Calculate the probability that the player will achieve his objective. 


Question no. 70 


Let { Xn, n = 0,1,2... } be a branching process for which Xo = 1. Suppose 
that there are two types of individuals, say A and B. Every individual (of type 
A or B) can give birth (independently from the others) to descendants of type 
A or B according to the formula 


PINa =m, Ng =n] =1/9 Vm,ne {0,1,2} 


where Ny (respectively, Ng) is the number of descendants of type A (resp., 
B). 
(a) Calculate E[X, | Xı > 0). 


(b) Show that the probability of eventual extinction of the population is 
qo œ 0.15417. 


Section 3.3 


Question no. 71 

Let Y := min{ X1, X2}, where X; and Xz are two independent exponential 
random variables, with parameters A; and àz, respectively. We know (see 
p. 113) that Y ~ Exp(A; + Ag). Find the probability density function of the 
random variable Z := Y | {X1 < X2}. 


Remark. We can express Z as follows: Z := Y | {Y = X ,}. However, the 
random variables Y | {Y = X,} and X, are not identical, because 
PY <y|Y =X) = PX <y[Y =X] = Pix; <y|X1 < X] 
#4 P|X1 < y] 


Note that the events {X1 < y} and {X1 < X2} are not independent. 


Question no. 72 
Let X1,... , Xn be independent random variables having an exponential 
distribution with parameter A. 
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(a) Use the memoryless property of the exponential distribution to show that 


1 1 
P(X, > X2 +X] = 7 = Gt 


(b) Show, by mathematical induction, that 


P 





n 

1 

Xj > = once forn = 25 Jae 
k=2 


(c) From the result in (b), calculate the probability P[2Y > >~;_, Xk], where 
Y := max{Xı,... gn} 


Question no. 73 | 

A birth and death process having parameters A, = 0 and un = H, for 
all n > 0, is a pure death process with constant death rate. Find, without 
making use of the Kolmogorov equations, the transition function p;,;(t) for 
this process. 


Question no. 74 
Let Xı ~ Exp(A1) and X2 ~ Exp(A2) be two independent random vari- 
ables. Show that, for all x > 0, 


A 
PIX, < Xo | min{ X1, X2} = z] i PIX, s Xə] 7 Àl A 


Question no. 75 

A system is made up of three components placed in standby redundancy: 
at first, only component no. 1 is active and when it fails, component no. 2 
immediately relieves it. Next, at the moment when component no. 2 fails, 
component no. 3 becomes active at once. When the system breaks down, the 
three components are instantly replaced by new ones. Suppose that the life- 
time Tk of component no. k has an exponential distribution with parameter 
Ax, for k = 1,2,3, and that the random variables 7;, T2, and 73 are indepen- 
dent. Let the number of the component that is active at time ¢t be the state 
of the system at this time instant. Write the Kolmogorov backward equations 
for this system. 


Question no. 76 

A university professor cannot receive more than two students at the same 
time in her office. On the day before an exam, students arrive according to a 
Poisson process with rate À = 3 per hour to ask questions. The professor helps 
the students one at a time. There is a chair in her office where a person can 
wait his or her turn. However, if a student arrives when two other students 
are already in the professor’s office, then this student must come back later. 
We suppose that the time that the professor takes to answer the questions of 
an arbitrary student is an exponential random variable with mean equal to 15 
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minutes, independently from one student to another. If we consider only the 
days preceding an exam, calculate (with the help of the limiting probabilities) 


(a) the average number of students in the professor’s office, 


(b) the proportion of time, on the long run, when the professor is not busy 
answering questions. 


(c) If the professor spent twice more time, on average, with each student, what 
would be the answer in (b)? 


Question no. 77 

Let {X(t),t > 0} and {Y(¢),t > 0} be two continuous-time indepen- 
dent Markov chains. We consider the two-dimensional stochastic process 
{(X(t), Y(t)),¢ > 0}. Find the parameters u(j,4), P,k), (j,k) 24 PG,k), (4,1) OF 
this process. 


Question no. 78 

We consider a pure birth process for which, when there are n individuals 
in the population, the average time (in hours) needed for a birth to occur is 
equal to 1/n, for n > 0. 


(a) Knowing that at time t there are two individuals in the population and 
that at time t + 1 there are still two, what is the probability that the next 
birth will take place between t + 2 and t + 3? 


(b) If, at the origin, the population is composed of a single individual, what 
is the probability that there will be exactly four births during the first two 
hours? 


Question no. 79 

A factory has m machines. Each machine fails at an exponential rate p. 
When a machine fails, it remains down during a random time having an 
exponential distribution with parameter A. Moreover, the machines are in- 
dependent from one another. Let X(t) be the number of machines that are 
in working order at time t > 0. It can be shown that the stochastic process 
{X(t),t > 0} is a birth and death process. 


(a) Find the birth and death rates of the process {X(t),t > 0}. 
(b) Show that 
lim P[X(t) =n] = P [B (m x5) = n! 


too 

forn = 0 les m: 
Question no. 80 

Let {X(t),t > 0} be a pure birth process such that A; = jA, for j = 
0,1,..., where \ > 0. We suppose that X (0) = 1. 
(a) Let Tn := min{t > 0: X(t) =n (> 1)}. That is, Tn is the time needed for 
the number of individuals in the population to be equal to n. Show that the 
probability density function of Ta is given by 
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(b) Let N(t) be the number of descendants of the ancestor of the population 
at time t, so that N(t) = X(t) — 1. Suppose that the random variable 7 has 
an exponential distribution with parameter u. Show that 


PIN(r) =n] = ŠB (& +1241) for = Bev 


where B(-,-) is the beta function defined by 


1 
B(z,y) =| #77 (1 —t)? dt 


for x,y € (0,00). 


Remark. We can show that 


P(x) PCy) 
B(x, y) = ——— 
(x,y) Fea) 
Question no. 81 
A birth and death process, {X (t), t > 0}, has the following birth and death 
rates: 
Ay x. torn =0,1. and p,=nyw. forn=1,2 


Moreover, the capacity of the system is equal to two individuals. 

(a) Calculate, assuming that A = u, the average number of individuals in the 
system at a time instant ¢ (large enough), given that the system is not empty 
at this time. 

(b) Calculate the probability that the process will spend more time in state 
0 than in state 1 on two arbitrary visits to these states. 

(c) Suppose that jz; = 0 and that, when X(t) = 2, the next state visited will 
be 0, at rate 2u. Write the balance equations of the system, and solve them 
to obtain the limiting probabilities. 


Question no. 82 
Let {X (t), t > 0} be a birth and death process whose rates A, and fn are 
given by 


Ke = la SNe Tore =0, 1. 2.845 


We set p(t) = P[X(t) = k], for all k € {0,1,2,...} and for all t > 0. That 
is, p(t) denotes the probability that the process will be in state k at time t. 
Suppose that p;(0) = 1. It can be shown that 


dt (At)E-1 


— e Cee fi A 
fa, OOP Pee 


po(t) 
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(a) Calculate EIX (t) | X(t) > O]. 

Indication. We have E|X(t)| = 1. 

(b) Calculate the limiting probabilities m; and show that they satisfy the 
balance equations of the process. 

(c) Use the Kolmogorov backward equation satisfied by p1,o(t) to obtain 
p2,o(t). 

Indication. We have pı o(t) = po(t) above. 


Question no. 83 

We consider a system composed of three components placed in parallel 
and operating independently. The lifetime X; (in months) of component i 
has an exponential distribution with parameter À, for i = 1,2,3. When the 
system breaks down, the three components are replaced in an exponential 
time (in months) with parameter u. Let X(t) be the number of components 
functioning at time t. Then {X(t),t > 0} is a continuous-time Markov chain 
whose state space is the set {0, 1, 2,3}. 
(a) Calculate the average time that the process spends in each state. 
(b) Is the process {X(t),t > 0} a birth and death process? Justify. 
(c) Write the Kolmogorov backward equation for po, o(t). 


(d) Calculate the limiting probabilities of the process if A = p. 


Question no. 84 

Let {N(t),t > 0} be a counting process (see p. 231) such that N(0) = 0. 
When the process is in state j, the next state visited will be j + 1, for all 
j > 0. Moreover, the time 7; that the process spends in state j has the 
following probability density function: 


fr;(s) = 2(7 + 1)As eTit? for s > 0 


Finally, the 7;’s are independent random variables. 
Now, consider the stochastic process {X (u), u > 0} defined by 


Oifu<ré 
oa 7 if De T <us ae es 


ORS a 

(a) Show that the stochastic process {X(u),u > 0} is a continuous-time 
Markov chain. 

(b) Calculate P[X (2) = 1]. 

(c) Calculate, if they exist, the limiting probabilities 7; of the stochastic pro- 
cess {X(u),u > 0}. 

Question no. 85 


We consider the particular case of the gambler’s ruin problem (see p. 101) 
for which k = 4 and p = 1/2. Suppose that the length T (in minutes) of 
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a play (the outcome of which is the player’s winning or losing $1) has an 
exponential distribution with mean equal to 1/2. Moreover, when the player’s 
fortune reaches $0 or $4, he waits for an exponential time S (in hours) with 
mean equal to 2 before starting to play again, and this time is independent of 
what happened before. Finally, suppose that the player has $1 when he starts 
to play again if he was ruined on the last play of the preceding game and that 
he has $3 if his fortune reached $4 on this last play. 

Let X(t), for t > 0, be the player’s fortune at time t. The stochastic process 
{X (t), t > 0} is a continuous-time Markov chain. 


(a) (i) Is the process {X(t),t > 0} a birth and death process? If it is, give its 
birth and death rates. If it’s not, justify. 

(ii) Answer the same question if the player always starts to play again 
with $1, whether his fortune reached $0 or $4 on the last play of the previous 
game. 


(b) (i) Write, for each state 7, the Kolmogorov backward Ss satisfied 
by the function po,;(t). 
(ii) Use the preceding result to obtain the value of the sum m y j=0 Po, j (É). 


(c) Calculate the limiting probabilities of the process {X (t),t > 0}, for all 
states 7. 


Question no. 86 
Let {Xn,n = 0,1,...} be a (discrete-time) Markov chain whose state 
space is {0, 1,2} and whose one-step transition probability matrix is 


Q (1 — a)/2 (1 — a) /2 
P= | (1-a)/2 a (1 —a)/2 
2X1 -a)/3 (1-a)/3 a 


where a € [0,1]. Suppose that the process spends an exponential time with 
parameter À in state i before making a transition, with probability pi j, to 
state j, for 2,7 € {0,1,2}. Let X(t) be the position, that is, the state in which 
the process is, at time t. 


(a) For what value(s) of a is the stochastic process {X (t), t > 0} a continuous- 
time Markov chain? Justify. 


(b) For the value(s} of a in (a), calculate, assuming they exist, the limiting 
probabilities 7;, for 7 = 0,1, 2. 


Question no. 87 
In the preceding question, suppose that the transition matrix P is instead 
the following: 


a BPl-a-®B 
P= y 0 1-y¥ 
1-68 0 


where a, ĝ, and y € (0, 1]. 
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(a) For what values of a, 8, and y is the stochastic process {X(t),t > O} a 
birth and death process? Justify and give the value of the parameters vj, Aj, 
and uj, for 7 = 0,1, 2. 

(b) For the values of a and 8 found in (a), and for y € (0,1), calculate, 
assuming they exist, the limiting probabilities 7;, for all 7. 


Question no. 88 

Let {X(t),t > 0} be a (continuous-time) stochastic process whose state 
space is the set {0,1}. Suppose that the process spends an exponential time 
with parameter A in a state before making a transition to the other state, 
where A is a discrete random variable taking the values 1 and 2 with proba- 
bility 1/3 and 2/3, respectively. 
(a) Is the stochastic process {X(t),t > 0} a continuous-time Markov chain? 
Justify. 


(b) Suppose that X(0) = 0. Let Tọ be the time that the process spends in 
state 0 before making a first transition to state 1. Calculate the probability 
P|A=1|7 < I. 


Question no. 89 

Suppose that the continuous-time stochastic process {X(t),t > 0}, whose 
state space is {0,1}, spends an exponential time with parameter 1 in a state 
the first time it visits this state. The second time it visits a state, it stays there 
an exponential time with parameter 2. When both states have been visited 
twice each, the process starts anew. 


(a) Is the stochastic process {X (t),¢ > 0} a birth and death process? Justify. 


(b) Let N be the number of visits to state 0 from the initial time 0, and let 
To be the time that the process spends in state 0 on an arbitrary visit to this 
state. Calculate approximately P[N is odd | To < 1] if we assume that the 
most recent visit to state 0 started at a very large time t. 


Question no. 90 

A system is composed of three components operating independently. Two 
active components are sufficient for the system to function. Calculate the 
failure rate of the system if the lifetime 7; of component 7 has an exponential 
distribution with parameter À = 2, for i = 1, 2,3. 


Question no. 91 
Let {Ni (t),¢ > 0} and {N2(t),t > 0} be two independent Yule processes, 
with rates A, = nı and A, = nbz, for n = 0,1,..., respectively. We define 


X(t) == N (t) + No(t) for t > 0 


(a) For what values of the constants 0; and 69 is the stochastic process 
{X(t),¢ > 0} a continuous-time Markov chain? Justify and give the value 
of the parameters vn of this process. 


(b) For the values of 6; and @2 found in (a), calculate p;,;(t), for j >i > 1. 
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Question no. 92 
We define 
X(t) = |Ni(t) — Noa(t)| for t > 0 
where {N,(t),t > 0} and {N2(t),¢ > 0} are two independent Poisson pro- 
cesses, with rates Ay = Ao = A. 


(a) Show that {X(t),t > Ki is a birth and death process, and give the rates 
Àn and Hn, for n = 0,1,. 


(b) Calculate, if they exist, the limiting probabilities 7;, for j = 0,1,... 
Question no. 93 


Calculate P|X1 < X2 < X3] if X1, Xo, and X3 are independent random 
variables such that X; ~ Exp(,,;), for i = 1, 2,3. 


Question no. 94 
Let {X(t),t > 0} be a birth and death process whose state space is the 
set {0,1,2} and for which 


ào =À, A= pu =A/2, and p2=A 


We consider two independent copies, {X; (t), t > 0} and {X2(t), t > 0}, of this 
process, and we define 


Y(t) = |Xi(t) — X(t)| for t > 0 


We can show that {Y(t),t > 0} is also a birth and death process. 
(a) Give the birth and death rates of the process {Y (t),t > 0}. 


(b) Calculate the expected value of the random variable Y (t) after two tran- 
sitions if X,(0) = X2(0) = 0. 
(c) Calculate the limiting probabilities of the process {Y (t), t > 0}. 


Question no. 95 
We consider a birth and death process, {X(t),t > 0}, whose state space 
is the set {0,1,2,...} and whose birth and death rates are given by 


An =nA and p,=nu forn=0,1,... 
where A, u > 0. Suppose that X(0) = i € {1,2,...}. Calculate E[.X (é)]. 


Indication. We can use the Kolmogorov equations. 


Question no. 96 
The rates Àn and un of the birth and death process {X(t),t > = 0}, whose 
state space is the set {0,1,...,c}, are 


An =(c—n)A and pn = np 


for n = 0,1,...,c. Suppose that X(0) = k e {0,1,...,c}. Calculate the 
function Pk e(t). 
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Indication. In the case of a continuous-time Markov chain defined on the set 
{0,1}, and for which 


NG = As A, = 0, uo = 0, and by = pb 


we have (see p. 131) 





Po,o(t) utr u+À 
and F i 
Pye, 2 a pe Ae 
pio(t) IA EA 


Question no. 97 
Let {X (t), t > 0} be a birth and death process for which 





1 
Am torn =0 less anid emn forns l 2e 
n+l 


and let tp > 0 be the time instant at which the first birth occurred. 


(a) Suppose that we round off the time by taking the integer part. Calculate 
the probability that the first event from tọ will be a birth. 

Indication. If X ~ Exp(A), then 1 + int(X) ~ Geom(1 — e~*), where int 
denotes the integer part. 


(b) What is the probability that there will be at least two births among the 
first three events from to? 


(c) Calculate, if they exist, the limiting probabilities of the process. 


Question no. 98 

The lifetime of a certain machine is a random variable having an expo- 
nential distribution with parameter A. When the machine breaks down, there 
is a probability equal to p (respectively, 1 — p) that the failure is of type I 
(resp., II). In the case of a type I failure, the machine is out of use for an 
exponential time, with mean equal to 1/y time unit(s). To repair a type I 
failure, two independent operations must be performed. Each operation takes 
an exponential time with mean equal to 1/p. 


(a) Define a state space such that the process {X(t),t > 0}, where X(t) 
denotes the state of the system at time t, is a continuous-time Markov chain. 


(b) Calculate, assuming the existence of the limiting probabilities, the prob- 
ability that the machine will be functioning at a (large enough) given time 
instant. 


Question no. 99 


A person visits a certain Web site according to a Poisson process with 
rate À per day. The site in question contains a main page and an internal 
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link. The probability that the person visits only the main page is equal to 
3/4 (independently from one visit to another). Moreover, when she clicks on 
the internal link (at most once per visit), the probability that she will return 
(afterward) to the main page is equal to 1/2. We define the states 


0: the person is not visiting the site in question 

1: the person is visiting the main page (coming from outside the site) 
2: the person is visiting the internal link 

3: the person is visiting the main page, after having visited the link 


Let 7 be the time (in hours) spent in state k, for k = 0,1,2,3. We assume 
that the random variables Tę are independent and that Tk has an exponential 
distribution with parameter vg, for k = 1,2, 3. 

Remark. We suppose that when the process is in state 3, the internal link is 
highlighted and that this highlighting is removed when the person leaves the 
site. 

Let X(t) be the state the process is in at time t > 0. We can show that 
{X(t),t > 0} is a continuous-time Markov chain. 


(a) Give the probabilities p; ; of the process. 


(b) Is the process {X(t),t > 0} a birth and death process? If it’s not, is it 
possible to rename the states so that {X(t),¢ > 0} becomes a birth and death 
process? Justify. 


(c) Calculate the average time spent on the site in question on an arbitrary 
visit, given that 7; = 1/4 for this visit. 


(d) Calculate the limiting probabilities 7;. 


A 


Diffusion Processes 


4.1 The Wiener process 


We already mentioned the Wiener process twice in Chapter 2. In this section, 
we will first present a classic way of obtaining this process from a random 
walk. Then we will give its main properties. 


Consider the discrete-time Markov chain {X,,n = 0,1,...} whose state 
space is the set of all integers Z := {0,+1,+2,...} and whose one-step tran- 
sition probabilities are given by 


Pii+1 = Pii-1 = 1/2 forallz eZ (4.1) 


This Markov chain is a symmetric random walk (see p. 48). A possible interpre- 
tation of this process is the following: suppose that a particle moves randomly 
among all the integers. At each time unit, for example, each minute, a fair coin 
is tossed. If “tails” (respectively, “heads”) appears, then the particle moves 
one integer (that is, one unit of distance) to the right (resp., left). 

To obtain the stochastic process called the Brownian motion, we accelerate 
the random walk. The displacements are made every 6 unit of time, and the 
distance traveled by the particle is equal to € unit of distance to the left or to 
the right, where, by convention, 6 > 0 and e > 0 are real numbers that can be 
chosen as small as we want. As the Wiener process is a continuous-time and 
continuous-state process, we will take the limit as 6 and e decrease to 0, so that 
the particle will move continuously, but will travel an infinitesimal distance on 
each displacement. However, as will be seen subsequently, we cannot allow the 
constants ô and € to decrease to 0 independently from each other; otherwise, 
the variance of the limiting process is equal either to zero or to infinity, so 
that this limiting process would be devoid of interest. 

We denote by X(t) the position of the particle at time t, and we suppose 
that X(0) = 0. That is, the particle is at the origin at the initial time. Let 
N be the number of transitions to the right that the particle has made after 


174 4 Diffusion Processes 


its first n displacements. We can then write that the position of the particle 
after nô unit(s) of time is given by 


X (nd) = (2N -nje (4.2) 


Note that if all the displacements have been to the right, so that N = n, 
then we indeed have that X(néd) = ne. Similarly, if N = 0, then we obtain 
X (nd) = —ne, as it should be. 


Remark. Since the particle only moves at time instants 6, 26, ..., we may 
write that its position at time t is given by 


X(t) = X([t/ô] ô) for all t > 0 (4.3) 


where | | denotes the integer part. 


Because the tosses of the coin are independent, the random variable N has 
a binomial distribution with parameters n and p = 1/2. It follows that 


E|X(n6)] == (2x B-n)€=0 (4.4) 
and 
V[X(nd)] = 4 2 VIN] =4 è x z =n eè (4.5) 


If we first let 6 decrease to 0, then the random walk becomes a continuous- 
tıme process. However, given that 


t 
VIX llenni = z X e? (4.6) 
we find that € must tend to 0 at the same time as 6; otherwise, the variance 
of X(t) will be infinite. Actually, to obtain an interesting limiting process, 6 
and e? must decrease to 0 at the same speed. Consequently, we assume that 
there exists a constant o (> 0) such that 


exoV6 <> e= (4.7) 


Thus, when we let 6 decrease to 0, we obtain a process that is also with 
continuous-state space and for which 


E([X(t)] =0 (4.8) 
and 
Vixa] ot vt>0 (4.9) 


Remark. By choosing e = avô, we directly have that V[X(t)] = 07 t, for all 
t > 0. That is, the variance of X(t) is actually equal to o7t for any positive 
value of 6, and not only in the limit as 6 | 0. 
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From the formula (1.113), we may write that 
P[X(nd) < a] ~ P[N(0, ne?) < x] (4.10) 
from which we deduce that 
P[W (t) < a] = P[N(0,07t) <a] (4.11) 
where 


W(t) = lim X(t) (4.12) 


That is, the random variable W(t) has a Gaussian distribution with zero 
mean and variance equal to o? t. This result is the essential characteristic of 
the Wiener process. Moreover, since a random walk is a process with inde- 
pendent and stationary increments (see p. 50), we can assert that the process 
{W (t), t > 0} has these two properties as well. 


Based on what precedes, we now formally define the Wiener process. 


Definition 4.1.1. A stochastic process {W(t),t > 0} is called a Wiener 
process, or a Brownian motion, if 


i) W(0) = 0, 
ii) {W(t),t > 0} has independent and stationary increments, 
iii) W(t) ~ NO,o?t) Vt>0. 


Remarks. i) The name Brownian motion is in honor of the Scottish botanist 
Robert Brown.! He observed through a microscope, in 1827, the purely ran- 
dom movement of grains of pollen suspended in water. This movement is due 
to the fact that the grains of pollen are bombarded by water molecules, which 
was only established in 1905, because the instruments Brown had at his dis- 
posal at the time did not enable him to observe the water molecules. The 
Brownian motion and the Poisson process (see Chapter 5) are the two most 
important processes for the applications. The Wiener process and processes 
derived from it are notably used extensively in financial mathematics. 


ii) Let 
Bi) <= me (4.13) 


We have that V|B(t)] = t. The stochastic process {B(t),t > 0} is named 
a standard Brownian motion. Moreover, if we sample a standard Brownian 
motion at regular intervals, we can obtain a symmetric random walk. 


1 Robert Brown, 1773-1858, was born in Scotland and died in England. He was a 
member of the Royal Society, in England. 
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iii) The Wiener process has been proposed as a model for the position of the 
particle at time t. Since the distance traveled in a time interval of length ô 
is proportional to vô, we then deduce that the order of magnitude of the 
velocity of the particle in this interval is given by 


vô = a — oo whend|0 (4.14) 


ô Vó 
However, let V(t) be the average velocity (from the initial time) of the 
particle at time t > 0. It was found experimentally that the model 
W(t 
V(t) := al for t > 0 (4.15) 
was very good for values of t large enough (with respect to 6). We will see 
another model, in Subsection 4.2.5, which will be appropriate for the velocity 
of the particle even when t is small. 


iv) We can replace condition iii) in Definition 4.1.1 by 
W(t+s)—W(s)~N(0,o7t) Vs,t>0 | (4.16) 


and then it is no longer necessary to assume explicitly that the process 
{W(t),t > 0} has stationary increments, since it now follows from the new 
condition iii). 
v) Let 

W*(t) :=Wi(t)+e (4.17) 


where c is a real constant, which is actually the value of W*(0). The process 
{W*(t),t > 0} is called a Brownian motion starting from c. We have that 
W*(t) ~ N(c,o7t), Vt > 0. We could also consider the case when c is a 
random variable C independent of W(t), for all t > 0. Then we would have 


E(W*(t)] = E[C] and V[W*(t)] = o°t+ V{C] (4.18) 


vi) Wiener proved the following very important result: W(t) is a continuous 
function of t (with probability 1). Figure 4.1 shows a (simplified) example of 
the displacement of a particle that would follow a Brownian motion. In reality, 
the trajectory of the particle would be much more complicated, because there 
should be an infinite number of changes of direction of the particle in any 
interval of finite length. We can thus state that the function W(t) is nowhere 
differentiable (see, however, Section 4.3). 


In general, it is very difficult to explicitly calculate the kth-order density 
function (see p. 49) of a stochastic process. However, in the case of the Wiener 
process, {W(t),t > 0}, we only have to use the fact that this process has 
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W(t) 


Fig. 4.1. (Simplified) example of the Brownian motion of a particle. 


independent and stationary increments. We can thus write, for t} < t2 < 
ea Cpe that 





k 
Fie SW tossa] =f wrt) I] f(w; —wj-1it; —tj-1) (4.19) 
j=2 
where 
ae 
f(w;t) = FET exp ia] for all w € R (4.20) 


is the density function of a random variable having a Gaussian N(0, 07) dis- 
tribution. Indeed, we have 


k 
(| (W) = wy} (4.21) 
j=l 


k 
= {W (t1) = wid) 4 1) {Wts) — W(tj-1) = wj — wyj-1} 
j=2 


and the random variables W (t1), W (t2) — W (ti), ... are independent and 

all have Gaussian distributions with zero means and variances given by Gti 
2 

g (t2 — ty), TEN 


Remark. We deduce from what precedes that the Wiener process is a Gaussian 
process (see Section 2.4). It is also a Markovian process, because it is the limit 
of a Markov chain. 


To calculate the autocovariance function (see p. 49) of the Wiener process, 
note first that 
Cov[X,Y + Z| = E[X (Y + Z)] — E[X]E[Y + Z] 
= {E[XY] — E[X]E[Y]} + {E[X 2] — E[X]E(Z]} 
= Cov|X, Y] + Cov|X, Z] (4.22) 
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Then we may write that 


Cwlt,t+s) = Cov[W(t),W(t+s)| 
= Cov{W(t), W(t) + W(t+s) —W(t)] 
= CoviW(t),W(t)] + Cov[W (t), W(t + s) — W (t)] 
ind. incr: Ooy[W (t), W(t)] = V[W(t)] = o°t (4.23) 


for all s,t > 0. This formula is equivalent to 
Cw(s,t) = o° min{s,t} for all s,t >0 (4.24) 


Remarks. i) The random variables W(t) and W(t + s) are not independent, 
because the intervals [0, t] and [0, t+ s] are not disjoint. Thus, the larger W(t) 
is, the larger we expect W(t + s) to be. More precisely, we have 


W(t+s)|W(t)~N(W(t),0%s) Vs,t>0 (4.25) 


li) Since the function Cw (t,t + s) is not a function of s alone, the Wiener 
process is not stationary, not even in the wide sense (see p. 52). As we already 
mentioned, the notion of processes with stationary increments and that of 
stationary processes must not be confounded. 


iii) Since E[W(t)] = 0, we also have 
Rw(s,t) = o° min{s,t} for all s,t >0 (4.26) 


where Ry(-,-) is the autocorrelation function of the process {W(t),t > 0} 
(see p. 49). 


Example 4.1.1. If the random variables W(t) and W(t + s) were independent, 
we would have W(t) + W (t + s) ~ N(0,a?(2t + s)), which is false. Indeed, we 
may write that 


W(t)+W(t+s)=2W(t)+|W(t+s) -W(t)] =X+Y 


where X := 2W(t) and Y := W(t + s) — W(t) are independent random 
variables, because the Wiener process has independent increments. Moreover, 
we have 


and | 
V[X})=4V[W(t)] =407t and V[Y]=V[W(s)] =07s 


(using the fact that the Wiener process also has stationary increments). We 
thus have 
W(t) + W(t+s) ~ N(0,07(4t + s)) 


which follows directly from the formula (1.108) as well. 
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Remark. As the increments of the Wiener process are stationary, the random 
variables W(t + s) — W(t) and W(s) (= W(s) — W(0)) have the same dis- 
tribution. However, this does not mean that W(t + s) — W(t) and W(s) are 
identical variables. Indeed, suppose that t = 1, s = 1, and W(1) = 0. We 
cannot assert that W(2) -W(1) = W(2) — 0 = 0, since P[W(2) = 0] = 0, by 
continuity. 


Given that the Brownian motion is Gaussian and that a Gaussian process is 
completely determined by its mean and its autocovariance function (see p. 59), 
we can give a second way of defining a Brownian motion: a continuous-time 
and continuous-state stochastic process, {X(t),¢t > 0}, is a Brownian motion 
if 
i) X(0)=0, 

ii) {X (t), t > 0} is a Gaussian process, 

iii) E[X (t) = 0, 

iv) Cx(s,t) = Cov[X(s), X(t)] = o? min{s,t} Y s,t > 0, where o > Qisa 
constant. 

It is generally easier to check whether the process {X(t),t > 0} possesses 
the four properties above, rather than trying to show that its increments are 
or are not independent and stationary. This second definition of the Brownian 
motion is particularly useful when {X(t),t > 0} is some transformation of a 
Wiener process {W(t),t > 0}. As we saw in Section 2.4, any affine transfor- 
mation of a Gaussian process is also a Gaussian process. That is, if 


X(t) = aW (t) + co (4.27) 


where co and cı Æ 0 are constants, then {X(t),t > 0} is a Gaussian process. 
Moreover, if we only transform the variable t, for example, if 


X(t) = W(t?) (4.28) 


then the process {.X(t),t > 0} is also a Gaussian process. 


Remark. We could drop the first condition above if we accept that a Brownian 
motion can start from any point wo € R. Similarly, we could replace the third 
condition by E[X(t) = ut, where pu is a real constant. In this case, the 
stochastic process {X (t), t > 0} would be a Brownian motion with drift u (see 
Subsection 4.2.1). 


Example 4.1.2. Let {W(t),t > 0} be a Brownian motion. We set 
X(0)=0 and X(t)=tW(l1/t) ift>0 


At first sight, the stochastic process {X(t),t > 0} does not seem to be a 
Wiener process. However, we have 
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EX (t) = tE[W(1/t)] =t-0=0 ift>0 
because E[W (t)] = 0, for any value of t > 0, and then 


Cx(s,t) = E[X(s)X(t)] -0x0 
= Els W(1/s)t W(1/t)] = st Cw(1/s, 1/t) 
= sto? min{1/s,1/t} = o° min{s, t} 


Moreover, we can assert that {X(t),t > 0} is a Gaussian process, because here 
X(t) is a linear transformation of W(1/t). Since X(0) = 0 (by assumption), 
we can conclude that {X(t),t > 0} is a Brownian motion having the same 
characteristics as {W(t),t > 0}. 


Remark. We must not forget that the variable ¢ is deterministic, and not 
random. Thus, we can consider it as a constant in the calculation of the 
moments of the process {X (t), t > 0}. 


Example 4.1.3. We define the stochastic process {X(t),t > 0} by 
X(t) = B(t) |{B(t) > 0} fort >0 


where {B(t),t > 0} is a standard Brownian motion. 
(a) Show that the probability density function of X(t) is given by 


fx(y(@) = 2 feaz) for x 20 


(b) Calculate (i) E[X(t)] and (ii) V[X(t)], for t > 0. 

(c) Is the stochastic process {X(t),t > 0} (i) Gaussian? (ii) wide-sense sta- 
tionary? Justify. | 

(d) Are the random variables X(t) and Y(t) := |B(t)| identically distributed? 


Solution. (a) We may write that 


fea (z) 


fx lz) = PIBO > 0] = 2 fg (x) forr>0 


because B(t) ~ N(0,t) = P[B(t) > 0] = 1/2. 
(b) (i) We calculate 


o 1/2 poo 
(a) 2 a2 sor (=) / T\ z? 
EIX = T ———e dz = — | — —— }e dx 
N | Vrt T 0 ( 4 


OO 


= — (Uwfr)? e7®/#] = (2t/n)/? fort > 0 
0 
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(ii) Notice first that 


E|B?(t)] = E[B?(t) | B(t) > 0|P[B(¢) > 0] + B[B?(t) | Bit) < 0] PIB) < 0l 


1/2 


Since E[B*(t)] = V[B(t)] = t and E[B?(t) | B(t) > 0] = E[B?(t) | B(t) < 0] 
(by symmetry and continuity), it follows that 

V(X (t) 2 E[B? (t) | B(t) > 0} — a t— = for t > 0 
T 


(c) (i) Since X(t) > 0 Y t > 0, the stochastic process {X (t), t > 0} is not 
Gaussian. 


(ii) It is not WSS either, because E[X(t)] is not a constant. 
(d) We have 


PIY (t) < y) “= Pl-y < Bit) < yl = Faw (y) — Faw (-y) 


=> fyy(y) = 2 feily) for y 20 


Thus, X(t) and Y(t) are identically distributed random variables. 


4.2 Diffusion processes 


Continuous-time and continuous-state Markovian processes are, under certain 
conditions, diffusion processes. The Wiener process is the archetype of this 
type of process. One way, which can be made even more rigorous, of defining 
a diffusion process is as follows. 


Definition 4.2.1. The continuous-time and continuous-state Markovian sto- 
chastic process {X(t),t > 0}, whose state space is an interval (a,b), is a 
diffusion process if 


lim =PUX(t +e) XQ) >6| X(t) =a] =0 (4.29) 


V ô >0 andV x € (a,b), and if its infinitesimal parameters defined by (see 
p. 63) 


m(x; t) = lim ~E|X(t +e)—X(t)| X(t) = x] (4.30) 
and 
v(z;t) = lim ~B(X(t +) —-X(t))2| X(t) =a) (4.31) 


are continuous functions of x and of t. 
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Remarks. i) The condition (4.29) means that the probability that the process 
will travel a distance greater than a fixed constant 6 during a sufficiently short 
period of time is very small. In practice, this condition implies that X(t) is a 
continuous function of t. 


ii) We assumed in the definition that the infinitesimal mean m(z;t) and the 
infinitesimal variance v(x;t) of the process exist. 


iii) The state space Sx (4) of the stochastic process {X (t), t > 0} may actually 
be any interval: [a,b], (a,], [a,b), or (a,b). Moreover, if the interval does 
not contain the endpoint a, then a may be equal to —oo. Similarly, we may 
have that b = œ if Sx(4) = (a,b) or [a,b). Finally, if Sxa) = [a,b], then the 
functions m(z;t) and v(x;t) must exist (and be continuous) for a < z < b 
only, etc. 


Example 4.2.1. In the case of the Wiener process, {W (t), t > 0}, we have that 
W(t+e) | {W(t) = w} ~ N(w,o7e), for all € > 0. We calculate 


lim PIW +e) - WO) > ô | W(t) = w] = lim = PIJN(0, o?e)| > 8) 


1 ô 2 ô 
upet E ote T i E 
lim a? Non > Sr lim 5 {1 (5) 


where 


P(x) = | gee ay (4.32) 


eal Zi 


is the distribution function of the N(0,1) distribution. We may write that 


S(x) = 5 (1 +erf (=) (4.33) 


where erf(-) is the error function. 
Finally, making use of the formula (4.33) and of the asymptotic expansion 


er yr d 
eta) =1- = {2 - So. 


which is valid for x > 1, we find that 
ee {1 -a (5z) = 6 
eļ0 € ae 


1 1 
im -E = W(t) = w] = lim-0 =0 
lim EW (é + €) — W(t) | W(t) = | = lim - 





Next, we have 
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and 
lim -EWC rös Wit)? | Wit) =u 


ji 
= lim -FE{Z*|, where Z ~ N(0,07e) 
€ € 


1 ] 
= lim -V[Z] = lim = 0*e = 0” 
EcejJO € el0 € 
Thus, we have m(x; t) = 0 and v(x; t) = o*. Because the infinitesimal param- 
eters of the Wiener process are constants, the functions m(x; t) and u(a;t) are 
indeed continuous. 


The most important case for the applications is the one when the diffusion 
process {X (t), t > 0} is time-homogeneous, so that the infinitesimal moments 
of {X(t),t > 0} are such that m(z;t) = m(x) and v(x; t) = v(x). We can then 
assert (see p. 64) that the process {Y(t),¢ > 0} defined by 


Y(t) =g|X(t)] fort >0 (4.34) 


where g is a strictly increasing or decreasing function on the interval [a,b] = 
Sx ¢) and such that the second derivative g’(x) exists and is continuous, for 
all x € (a,b), is also a diffusion process, whose infinitesimal parameters are 
given by 


my(y) = m(x)g'(x) + =u(a)g" (2) and vy (y) = v(z)[9'(x)/ (4.35) 


where the variable x is expressed in terms of y: x = g7™!(y) (the inverse 
function of g(x)). Moreover, we have that Sya) = [g(a), 9(®)] if g is strictly 
increasing, while Sy (4) = |g(b),g(a)| if g is strictly decreasing. 


Remarks. i) The function g must not be a function of the variable t. 


ii) We assume in what precedes that the process {X(t),t > 0} can move from 
any state x € (a,b) to any other state y € (a,b) with a positive probability. 
We say that {X(t),t > 0} is a regular diffusion process. Then the process 
{Y (t), > 0} is regular as well. 


4.2.1 Brownian motion with drift 


A first important transformation of the Wiener process is a generalization of 
this process. Let 


Y(t) = oB(t) + ut (4.36) 


where { B(t),t > 0} is a standard Brownian motion, and u and o Æ 0 are real 
constants. Note that in this case the function g would be given by 
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g(x, t) = ox + pt (4.37) 


Thus, we cannot use the formulas (4.35) to calculate the infinitesimal param- 
eters of the process {Y(t),¢ > 0}. However, we have 


EY (t) | Y (to) = yo] = yo + u(t — to) (4.38) 
and 
VIY) | ¥ (to) = yo] = 07 (t — to) (4.39) 


for all t > tọ. We then deduce from the formulas (2.58) and (2.59) that 


my(y)=p and vy(y)=o7 for ally (4.40) 


Remark. If we try to calculate the function my (y) from (4.35), treating t as 
a constant, we find that my(y) = 0, which is false, as we see in Eq. (4.40). 


Definition 4.2.2. Let {Y(t),t > 0} be a diffusion process whose infinitesimal 
parameters are given by my (y) = u and vy (y) = a7. The process {Y (t), t > 0} 
is called a Brownian motion (or Wiener process) with drift ju. 


Remarks. i) The parameter p is the drift coefficient, and o? is the diffusion 
coefficient of the process. The term parameter, rather than coefficient, is used 
as well. 


ii) If the random walk had not been symmetric in the preceding section, we 
would have obtained, under some conditions, a Wiener process with nonzero 
drift coefficient u. 


ili) Since the function f[B(t)] := oB(t) + ut is an affine transformation of the 
variable B(t) ~ N(0,t), we may write 


Y(t) ~ N(ut, ot) (4.41) 
or, more generally, 


Y(t) | {Y (to) = yo} ~ N(yo + u(t — to), o° (t —to)) Vt > to (4.42) 


Moreover, the process {Y (t),t > 0} is a Gaussian process having independent 
and stationary increments. It follows (with Y(0) = 0) that 


Ely (t + s)Y(#)] = EUN +s)—-Y(t)+ Y(t))Y(t)] 
mao EY (t +s) — Y (EY ()] + EY? E) 
sat ior BLY (s) BLY (t)] + EY? (t) 
= peepee (4.43) 


which implies that 


4.2 Diffusion processes 185 
Cy(t+s,t) = Cov[Y (t + s), Y()] = ElY(t+s)Y()] — E[Y (t + s) EY (| 
= pst +0o°t+ pE -— ult+s)ut=0ot Vs,t>0 (4.44) 


Thus, the Brownian motion with drift has the same autocovariance function 
as the Wiener process. 


iv) The conditional transition density function p(y, yo; t, to) (see p. 62) of the 
Brownian motion with drift coefficient u and diffusion coefficient o? satisfies 
the partial differential equation (see p. 64) 


2 92 
i ae i (4.45) 
ot Oy 2 Oy? 


as well as the equation 


ð 0 KO 
ay eg e i (4.46) 
Oto OYo 2 dyg 


We can check that 


P(Y, Yo; t, to) = soe DE, (4.47) 


————————--—- EX 
27a7(t — to) P| 2 a? (t — to) 
for y, yo € R and t > tp > 0. We have 
lim p(y, yo; t, to) = 6(y — yo) (4.48) 


which is the appropriate initial condition. 


4.2.2 Geometric Brownian motion 


A diffusion process that is very important in financial mathematics is obtained 
by taking the exponential of a Brownian motion with drift. 

Let {X(t),¢ > 0} be a Wiener process with drift coefficient u and diffusion 
coefficient a°. We set 


Y(t)=e* fort >0 (4.49) 


Since the function g(x) = e* does not depend on t, we deduce from (4.35) 
that 


a 1 
my (y) = pe + 50°e* = py + 50°y (4.50) 


and 


vy(y) = o° (e)? = oy? (4.51) 
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Definition 4.2.3. The stochastic process {Y (t),t > 0} whose infinitesimal 
parameters are given by my(y) = (u + 507)y and vy(y) = o7y? is called a 
geometric Brownian motion. | 


Remarks. i) The state space of the geometric Brownian motion is the interval 
(0,00), which follows directly from the definition Y(t) = e*. The origin is 
a natural boundary (see Section 4.4) for this process. It is used in financial 
mathematics as a model for the price of certain stocks. 


ii) Since Y(t) > 0, for all t > 0, the geometric Brownian motion is not a 
Gaussian process. For a fixed t, the variable Y (t) has a lognormal distribution 
with parameters pt and o7t. That is, 


1 (In y — yt)? 
= ee ee. 4.52 
Fro) = Ts ie l T for y > 0 (4.52) 


iii) We can generalize the definition of the geometric Brownian motion {Y (t), 
t > 0} by setting 


Y(t) =Y(0)e* (4.53) 
where Y(0) is a positive constant. As in the case of the Wiener process, the 
initial value Y(0) could actually be a random variable. 

iv) To obtain the conditional transition density function p(y, yo;t, to) of the 


process, we can solve the Kolmogorov forward equation 


Op 1 ,\ ð GOS es 
asf a 2 es = 0 4.54 


In the particular case when to = 0, we find that the solution of this partial 
differential equation that satisfies the initial condition 


lim p(y, Yo; t) = ô(y — yo) (4.55) 
is 
(Y, yo; t) : (in gy HEY (4.56) 
, Y0; t) = ——_ exp | -— n C . 


for y, Yo > 0 and t > 0. 


v) The geometric Brownian motion is appropriate to model the evolution of 
the value of certain stocks in financial mathematics when we assume that the 
ratios i 


X, X2 Xs 


a aa (4.57) 
o A 
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where Xo is the initial price of the stock and X;, is the price after k unit(s) 
of time, are independent and identically distributed random variables. 


We have used the expression diffusion process at the beginning of this 
subsection. By definition, this means that the geometric Brownian motion 
must be a Markovian process, which we now show. 


Proposition 4.2.1. The geometric Brownian motion is a Markovian process, 
whose conditional transition density function is time-homogeneous. 


Proof. We have, in the general case when Y(t) := Y(0)e*™, 


VERS ey Oe Cy HV Oreste 
= Y(t+s)=Y(t)ež®t)-X0) (4.58) 
We then deduce from the fact that the Wiener process has independent in- 
crements that Y(t + s), given Y(t), does not depend on the past. Thus, the 
process {Y (t), t > 0} is Markovian. 
We also deduce from the equation above and from the independent and 
stationary increments of the Wiener process that 
PIY (to +s) < y | Y (to) = yo] = P[Y (to)e¥ ©0+t9-¥ %0) < y | Y (to) = yo] 
=y Piet otee tto) < y/yo] 
= Ple*) < y/yo] (4.59) 


for all yo,y > 0 and for all tọ,s > 0, from which we can assert that the 
function p(y, yo; t, to) of the geometric Brownian motion is such that 


p(y: yo; t, to) = p(y, yort — to) O (4.60) 
Remarks. i) Equation (4.58) implies that 
Y(t + 8) £Y(t)e*©) (4.61) 


That is, the random variable Y (t + s) has the same distribution as Y (t)ež 09. 


ii) To prove that the conditional transition density function of the geometric 
Brownian motion is time-homogeneous, we can also check that the function 


[see (4.56)| 
re (4.62) 
y 


Py, Yo; t, to) = ——— m 
(Hy / 2ra2(t — to) 207(t — to) 


for y, yo > 0 and t > to > 0, satisfies the Kolmogorov forward equation (4.54), 
subject to 
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lim p(y, yo; t, to) = 6(y — yo) (4.63) 


From the formula Y(t) = e*), and making use of the formula for the 
moment-generating function (see p. 19) of a random variable X having a 
Gaussian N(j:,07) distribution, namely 


Mx(s) = Efle’*] = exp {su + 580° | (4.64) 
we find that the mean and the variance of Y (t) are given by 
1 
EY (t)] = exp l (1 + 52”) 7 (4.65) 
and | 
1 2 o*t 
V{Y(t)] = exp l (x + 5° ) at} (e — 1) (4.66) 


for all t > 0. Note that E[Y(0)] = 1, which is correct, since Y (0) = 1 (because 
X (0) = 0, by assumption). More generally, for all t > T, we have 


E pa = exp { (n + 50) (t — n} (4.67) 


V pa = exp { (x + 50°) 2(t — | Co — 1) (4.68) 


We also find, using the fact that the process is Markovian, that 


and 


EI) 1¥(9),05 8 <7] EYOIYO = Yep { (w+ 502) (t-7)} 
(4.69) 


and 


ElY°(t) | Y(s),0< s <7] = E[Y(t) | Y(7)] = ¥2(r) exp{2(u + 0?)(¢- 7)} 
(4.70) 


for all t > 7, from which we can calculate V[Y(t) | Y(s),0 < s <7]. 
Finally, we can write (see Ex. 4.1.1) that if Y(0) = 1, then 


E{Y(t+s)¥(t)] = E eaae -E e] 
= Mx(1), where X ~ N(u(2t + s), o° (4t + 8)) 
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= exp ac +s) + 50° (at + )} (4.71) 
from which, V s,t > 0 we have 
Cy(t+s,t) = E[V(t+s)Y(t)] — BIY (t + s)JEIY (0) 


= exp f u(r +s) + 507 (dt + )} 


— exp i (: + 50°) (t+ s) exp f (: + 57°) r) 


= exp fuor + s) + 50 (at + s} — exp l (x + 50) (2t + s) 


= exp { (u n 50°) (2t + s} (e = 1) (4.72) 


Remarks. i) Contrary to the Wiener process, the geometric Brownian motion 
is not a process with independent and stationary increments. Indeed, 


Y(t + 8) — Y (8) = Y (0)e¥ 6+9 — Y (0je¥ 9 = Y (0)(e¥ (+9) — e*(8}) (4.73) 
does not have the same distribution as 
Y(t) — Y (0) =Y(0)e* — Y (0) = Y(0)(e* — 1) (4.74) 
for all s > 0. Moreover, the random variables 
Y(t+s)—Y(t)=Y(0)je*“) —y(o)eX = V(0)(e% "9 — eX) (4.75) 


and Y (t)—Y (0) are not independent. To justify this assertion, we can calculate 
the covariance of these variables. Since Y (0) is not random, we have 


CoviY (t+ s) — Y(t), Y(t) — Y(0)| 
= {EY (t + s)¥(t)] - Y(O)E[Y (t + s)] ~ EY? (t)] + YE (t)]} 
— {EY (t+ s)| - FIY LEY 6] - Y (0)} 
= Cov[Y (t + s), Y(t)] — VIY) (4.76) 
Using the formulas (4.66) and (4.72), we may write that 


Cov[Y (t + s) — Y (t), Y (t) — Y (0) 


= (e7 —1) lexp i (x + 507) (2t + »)} — exp i (u + 50°) 2} 
= e7 - 1) exp (x + 50°) 2) [exp { (x + 50°) s} = r (4.77) 


which is different from zero if s, t > 0, so that the random variables considered 
are not independent. 
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ii) If the drift coefficient of the Wiener process is equal to —350", then the 
corresponding geometric Brownian motion is such that [see the formula (4.69)| 


EIY (t) | ¥(s),0<s <7) =Y(r) (4.78) 


Thus, the process {Y (t),t > 0} is a martingale (see p. 102). We can always 
obtain a martingale from an arbitrary Wiener process, by setting 


1 
Y*(t) = Y(0) exp fxe — (x + 50°) r) = exp {> (x + 50°) t} Y (t) 
(4.79) 
where Y (t) is defined in (4.53). Indeed, we then have 


E[Y* (t) | Y*(s),0 < s < 7] 
Saf- (n+ tet) eh erorcen 


nef 
= exp { - (a+; i thy resp { (u+ 50°) -n} 


sye f- (x + 50°) r} =Y*(r) (4.80) 


N| =| 


bo 


r} EY (t) | Y(8),0 <8 <7] 


N 


iii) There exists a discrete version of the geometric Brownian motion used, in 
particular, in financial mathematics. Let Yp := lIn Xn, where Xn is the price 
of the shares of a certain company at time n € {0,1,...}. We assume that 


Yn = u + Yn-1 + En (4.81) 


where ji is a constant and the €,,’s are independent random variables. Then 
we have 


n 
Y,=nput+ > a+ Yo (4.82) 
i=1 


If we now assume that en ~ N(0,o7), for all n, and that Yo is a constant, we 
obtain 


EY, =nu+Yo and V[Yn] =n o? (4.83) 


The discrete-time and continuous-state processes {Y,,n = 0,1,...} and 
{X,,n = 0,1,...} are called a discrete arithmetic Brownian motion and a 
discrete geometric Brownian motion, respectively. 
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4.2.3 Integrated Brownian motion 


Definition 4.2.4. Let {Y(t),t > 0} be a Brownian motion with drift coeffi- 
cient u and diffusion coefficient a*, and let 


t 
Z(t) = Z(0) + f Y(s) ds (4.84) 

0 
The stochastic process {Z(t),t > 0} is called an integrated Brownian mo- 


tion. 


Proposition 4.2.2. The integrated Brownian motion is a Gaussian process. 


Proof. First, we use the definition of an integral as the limit of a sum: 


Z(t) = Jin, SOY ¥ (5 +) (4.85) 


Since the Wiener process is a Gaussian process, the Y(tk/n)’s are Gaussian 
random variables. From this, it can be shown that the variable Z(t) has 
a Gaussian distribution and also that the random vector (Z(t1), Z(t2),.-. , 
Z(tn)) has a multinormal distribution, for all t1,t2,... ,t, and for any n, so 
that the process {Z(t),t > 0} is Gaussian. O 


Remark. We can write that 


Z(tk) = (Z(tk) — Z(tk-1)) + (Z(tk-1) — Z(tk-2)) +- 


+ (Z(t2) — Z(t1)) + Z(t1) (4.86) 
for k = 2,...,n, where we may assume that tı < to <---<t,. We have 
tk 
Z (tk) — Z(tk-1) = J Y (s) ds (4.87) 
tk—1 


However, even though the increments of the Wiener process are independent, 
the random variables (Z (t2) — Z(t1)) and Z(t,), etc., are not independent, as 
will be seen further on. Consequently, we cannot proceed in this way to prove 
that the integrated Brownian motion is Gaussian. 


Assuming that Z (0) is a constant, we calculate 
t t 
EIZ(®] = Z(0) + | E(Y(s)|ds = Z(0) + | (¥ (0) + us) ds 
0 0 
pt? 
= Z(0) + ¥ (t+ 5 (4.88) 


For the sake of simplicity, suppose now that {Y(t),t > 0} is a standard 
Brownian motion (starting from 0) and that Z(0) = 0. Making use of the 
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formula (see p. 178) E[Y(s)Y (t)] = min{s,t} (because E|Y(t)] = 0 and o = 
1), we may write that 


ElZ(t + s)Z(t)] = | / ara di | ¥ (o) de] 
-E| [ 5 E Yl) dude 
=f | E BLY (u)¥(w)] dude 
5 [ { a Pen eee J  min{u2} du] die 
= f| {f uaus f vau} dv 
= [ {F +ut+s-u)} dv 


=~? (5 4 5) (4.89) 


From the previous formula, we obtain, under the same assumptions as 
above, that 


Cov[Z(t + s) — Z(t), Z(t)] = E {[Z(t + s) — Z(H|Z(t)} — 0 
= E[Z(t + s)Z(t)| — E[Z*(¢)] 


=#(f4+5)- = 40 (4.90) 


which implies that Z(t + s) — Z(t) and Z(t) are not independent random 
variables V s > 0, and 


E |{Z(t +s) — Z(s)}?] = E[Z?(t + s)| + E[Z? (s)| — 2E[Z(t + s)Z(s)| 


3 3 3 
= t+) rE- (S45) = Fes 





+3 
# > = EZH] (4.91) 


from which we deduce that Z(t + s) — Z(s) and Z(t) are not identically dis- 
tributed random variables. 


We can generalize these results and state the following proposition. 


Proposition 4.2.3. The increments of the integrated Brownian motion are 
neither independent nor stationary. 
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Next, by definition of Z(t), we may write that 
t+s t t+s 
Z(t+s) = 2(0) + | ¥(r)dr = Z(0) + | Y(r)dr+ | Y(r)dr 
0 0 t 


= Z(t) + J” Y(T)dT (4.92) 


Proposition 4.2.4. The integrated Brownian motion is not a Markovian pro- 
cess. However, the two-dimensional stochastic process (Y (t), Z(t)) is Marko- 
vian. 


Proof. The value of the last integral above does not depend on the past if 
Y(t) is known, because the Wiener process is Markovian. Now, if we only 
know the value of Z(t), we do not know Y (t). Thus, for the future to depend 
only on the present, we must consider the two processes at the same time. O 


When {Y (t),t > 0} is a standard Brownian motion, the joint probability 
density function of the random vector (Y (t), Z(t)), starting from (yo, Zo), is 
given by 


p(y, 2, Yo, 20; t) 
_ PIY(t) € (y, y + dy], Z(t) € (z,z + dz] | Y (0) = yo, Z (0) = zo] 
dydz 


v3 2 6 6 
=E {zu — yo)” + z0 — yo)(z — žo — yot) — 73 (2 — zo — yot)” 
(4.93) 


for y, zZ, Yo, zo E€ R and t > 0. 


Remarks. i) The density function p(y, z, yo, zo;t) is a particular case of the 
joint conditional transition density function p(y, z, yo, zo; t, to). 


ii) To be more rigorous, we should take the limit as dy and dz decrease to 0 
above. However, this notation is often used in research papers. 


The function p(y, Z, yo, 20; t) is the solution of the partial differential equa- 
tion (namely, the Kolmogorov forward equation) 


ð ð 1 8? 
pta pn pE 4.94 
a Iaz” 9 Oy? os Ce 


which satisfies the initial condition 


a _ f 0 ify Ayo orzF % 
PLY, Z, Yo, 20;t = 0) = d(y — yo, Z — zo) = o if y = yo and z = zo (4.95) 
Indeed, taking the limit as t decreases to zero in the formula (4.93), we obtain 


lim p(y, Z, Y0, 20; t) = oly — Yo, + — zo) (4.96) 
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Remark. Actually, it is preferable to define the two-dimensional Dirac delta 
function by 


d(y — Yo, z — zo) = 0 if (y,z) Æ (Yo: 20) (4.97) 


and 
/ J ô(y — Yo, Z — 2) dy dz = 1 (4.98) 
We can then write that 


d(y — yo, Z — z0) = ô(y — yo)d(z — zo) (4.99) 


Note that we also obtain the formula above, even if we set 6(0,0) = 6(0) = œ, 
by assuming that 


6(0)6(z — zo) = d(y — yo)6(0) =0 if z Æ zo and y £ yo (4.100) 


The formula for the function p(y, z, yo, zo; t) is easily obtained by using the 
fact that it can be shown that the random vector (Y(t), Z(t)) has a bivariate 
normal distribution. Moreover, the bivariate normal distribution is completely 
characterized by the means, the variances, and the covariance of the two 
variables that constitute the random vector. Here, when {Y (t),t > 0} is a 
standard Brownian motion starting from Y (0) = yo, and Z(0) = zo, we deduce 
from what precedes that 


E\Y()|=y, VY] =t, and E[Z(t)] = zo + yot (4.101) 


Furthermore, since E{Y(s)Y (t)| = min{s,t} + y when Y(0) = yo, we find 
that the generalization of the formula (4.89) is 


t 
E[Z(t + s)Z(t)] = t? (G; + 5) + ygt’ + 26 + 2zoyot (4.102) 


so that 
VIZO] = EZ?) — {EO 
t’ i 
ig i yat? + za + 2zoyot — (zo + yot)” = 3 (4.103) 
Remark. The variance of Z(t) is thus independent of the initial values Z(0) 


and Y(0), as we could have guessed. 


Next, we calculate 


avoz)=£[y@ {a+ f Yeay 
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t 


= Yoo + f EY (Y (T) dr 


t 2 i 
; t 
= Yoz% + f (T -} yé) dT = Yozo + a + yst (4.104) 
0 


which implies that 


Cov[Y (t), Z(t)] = EY 1) Z) - BY C) EIZE) 
2 


t t? 
= 3 tY% + Yot — yo(zo + yot) = 5 (4.105) 


and 


Cov[Y(t),Z(t)] _ t/2 _ v3 
rovo e377 4% 


The expression for the function p(y, z, Yo, zo;t) is obtained by substituting 
these quantities into the formula (1.82). 





PY (t), Z(t) ‘= 


We can check that if we replace t by t — tp in the formula (4.93), then the 
function p(y, Z, yo, zo; t, to) thus obtained is also a solution of Eq. (4.94), such 
that 

lim ply, Z, Yo, Z0; t, to) = 5(y — Yo, Z — 20) (4.107) 
Consequently, we can state the following proposition. 


Proposition 4.2.5. The integrated Brownian motion is time-homogeneous. 
That is, 


P(Y, z, Yo, Z0;t, to) = ply, Z, yo, 20;t —to) Vt>tp >0 (4.108) 
Finally, in some applications, we consider as a model the Brownian mo- 


tion integrated more than once. For example, the doubly integrated Brownian 
motion 


D(t) := D(0) + [ Z(s)ds = D(0) + Z(0)t + [ f Y(r)drds_ (4.109) 


where {Y (t),t > 0} is a Wiener process with drift coefficient u and diffusion 
coefficient 07, is a Gaussian process for which 


E[D(t)] = D(0) + Z(0)t + J l, "(¥(0) + wr) dr ds 


= D(0)+ Z(0)t + Yoi + ge 


: (4.110) 


We also find that 
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45 
VID] = o? — (4.111) 
20 | 
Moreover, the three-dimensional process (Y (t), Z(t), D(¢)) has a multinor- 
mal (namely, trinormal or three-variate normal) distribution whose vector of 
means m and covariance matrix K (see p. 59) are given by 


Y(0)+ pt , 
m = Z(0)+ Y(0)t + u5 (4.112) 
2 3 
D(0) + Z(0)t+ Y(0)5 + ye 
and 
1 +t/2 t7/6 
K =o7t | t/2 t?/3 t3/8 (4.113) 


t?/6 t3/8 t4/20 


Example 4.2.2. In Example 4.1.1, we calculated the distribution of the sum 
W(t)+W(t+ s) by using the fact that the increments of the Wiener process 
are independent. Since the integrated Brownian motion does not have this 
property, we cannot proceed in the same way to obtain the distribution of Z := 
Z(t) + Z(t + s). However, we can assert that Z has a Gaussian distribution, 
with mean (if u = 0 and Y(0) = Z(0) = 0) 


E|Z| =0+0=0 
and with variance (if o? = 1) 


VIZ] = V[ Z + V[Z(t + s)] +2Cov[Z(t), Z(t + s)| 


t? 3 t 
BOR a (5+3) 





3 3 os 2 
At3 s’ 

= — 422s + ts? + — 
z t SE E 


4.2.4 Brownian bridge 


The processes that we studied so far in this chapter were all defined for values 
of the variable t in the interval [0, o0). However, an interesting process that 
is based on a standard Brownian motion, {B(t),t > 0}, and that has been 
the subject of many research papers in the last 20 years or so, is defined 
for t € [0,1] only. Moreover, it is a conditional diffusion process, because we 
suppose that B(1) = 0. Since it is as if the process thus obtained were tied at 
both ends, it is sometimes called the tied Wiener process, but most often the 
expression Brownian bridge is used. 
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Definition 4.2.5. Let {B(t),t > 0} be a standard Brownian motion. The 
conditional stochastic process {Z(t),0 <t <1}, where 


Z(t) := B(t) | {B(1) = 0} (4.114) 
is called a Brownian bridge. 


Remark. We deduce from the properties of the Brownian motion that the 
Brownian bridge is a Gaussian diffusion process (thus, it is also Markovian). 


Suppose that the standard Brownian motion {B(t),t > 0} starts in fact 
from B(0) = bọ and that B(s) = bs. Then, using the formula (which follows 
from the (independent and) stationary increments of the Brownian motion) 


f(t),B(s)(6,6s) feb) fa(s)— Bye) (Os — b) 


fa(t)|{B(s)=b.} (b) = jae Fata (be) (4.115) 

for 0 < t < s, we find that 
B(t) | {B(0) = bo, B(s) =b} vN (A BERD EAD) ytes) 
(4.116) 


Thus, when bọ = 0, s = 1, and b, = 0, we obtain that 
B(t) | {B(0) = 0, B(1) = 0} ~ N(0,t(1-t)) Vte (0,1) (4.117) 
so that 
EZO =0 Vte (0,1) (4.118) 
and 
EZ?) = V[Z(t)] =t(1 —t) Vt e (0,1) (4.119) 


With the help of the formulas (4.116), (4.118), and (4.119), we calculate 
the autocovariance function of the Brownian bridge, for 0 < t <7 < 1, as 
follows: 


Cz(t,T) = plete Z(T) = E|Z(t)Z(7)] -0x0 
B\E|Z(t)Z(r) | Z(T)]] = E[Z(r) E[Z() | 2(7)]] 
= E | z(r)22(7)| = 271-7) 
=t(l-r) if0<t<r<l (4.120) 


In general, we have 


Cz(t,7) = min{t,r}—tr ift,r € (0,1) (4.121) 
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We also deduce from the formula (4.116) that 


E|Z(t +e) — Z(t) | Z(t) = z] = -= (4.122) 





and 

E[(Z(t +) — Z(t))* | Z(t) = z] = e + o(€) (4.123) 
It follows that the infinitesimal parameters of the Brownian bridge are given 
by (see p. 63) 


£ 


£ 
Se 4,124 
1-—t 1—t ( ) 


7 = li — 
m(x; t) im 
and 
| o(€) 
v(t) = lim 1 oor 1 (4.125) 


forzé€ RandO<t <1. 


Remarks. i) The Brownian bridge thus has the same infinitesimal variance as 
the standard Brownian motion. However, its infinitesimal mean depends on t 
and tends to infinity (in absolute value if x Æ 0) as t increases to 1. 


ii) Since m(az;t) is a function of t, the Brownian bridge is not a time- 
homogeneous process. 


As we did in the case of the Brownian motion, we can use the fact that 
the Brownian bridge is a Gaussian process to give a second definition of this 
process as being a Gaussian process with zero mean and whose autocovariance 
function is given by the formula (4.121). 


Proposition 4.2.6. Let {B(t),t > 0} be a standard Brownian motion. The 
stochastic process {Z;(t),0 <t <1} defined by 


Z(t) = B(t) — tB(1) (4.126) 
1s a Brownian bridge. 


Proof. We can assert that {Z,(t),0 < t <1} is a Gaussian process. Moreover, 
we calculate 


E\2Z,(t)| = E[B(t)| — tE[B(1) =0-tx0=0 (4.127) 
and then, 


Cov[Z1(t), Z1(7)] = E[Z1()21(7)] = E[(B(t) - tB(1))(B(7) - rBQ))] 
= E[B(t)B(r)| —7 E[B(t)B(1)] —t EĻB(1)B(7)] 
+ trE[B? (1) 
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= min{t,r}-—7rxt—txrt+irxl 
= min{t, T} — tr (4.128) 


Since the mean and the autocovariance function of the Gaussian process 
{Z,(t),0 < t < 1} are identical to those of the Brownian bridge, we can 
conclude that it is indeed a Brownian bridge. 0 


Remark. In the same way, we can show that the process {Z(t),0 < t < 1} 
defined by 


t 


=) forO<t<l (4.129) 


Z2(t) =(1-t)B ( 
and Z2(1) = 0 is a Brownian bridge as well. 


4.2.5 The Ornstein—Uhlenbeck process 


As we mentioned in Section 4.1, the use of a Wiener process to model the 
displacement of a particle can be criticized. Indeed, for small values of the 
variable t, the Wiener process is not appropriate to represent the average ve- 
locity of the particle in the interval [0, t]. Moreover, the instantaneous velocity 
cannot be calculated, because the Brownian motion is nowhere differentiable. 

To remedy this problem, in 1930 Uhlenbeck? and Ornstein? proposed a 
model in which they supposed that it is the velocity of the particle that is 
influenced, in part, by the shocks with the neighboring particles. The velocity 
also depends on the frictional resistance of the surrounding medium. The effect 
of this resistance is proportional to the velocity. 

As in the case of the geometric Brownian motion, we can define the 
Ornstein—-Uhlenbeck (O.-U.) process from a Wiener process. Let {B(t),t > 0} 
be a standard Brownian motion. We set 


g? erat 


U(t)=e “B (E) for t> 0 (4.130) 


where a is a positive constant. It is thus a particular case of the transformation 
X(t) = g(t) B (F(t) (4.131) 


where f(t) is a nonnegative, continuous, and strictly increasing function, for 
t > 0, and g(t) is a (real) continuous function. Indeed, the exponential function 


* George Eugene Uhlenbeck, 1900-1988, was born in Indonesia and died in the 
United States. He was a physicist and mathematician whose family, coming from 
the Netherlands, returned there when he was six years old. His main research 
subject was statistical physics. He also worked on quantum mechanics and wrote 
two important papers on Brownian motion. 

3 Leonard Salomon Ornstein, 1880-1941, was born and died in the Netherlands. 
He was a physicist who worked on quantum mechanics. He applied statistical 
methods to problems in theoretical physics. 
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is continuous and (g? /2a)e?™ is nonnegative and strictly increasing, because 
a > 0. For any transformation of this type, the following proposition can be 
shown. 


Proposition 4.2.7. Under the conditions mentioned above, the stochastic 
process {X(t),t > 0}, where X(t) is defined in (4.131), is a Gaussian process, 
whose infinitesimal parameters are given by 


m(x; t) = ron and v(x;t) =g (O (t) (4.132) 


forx ER andt >Q. 


Remark. The process {X(t),t > 0} is Gaussian, because the transformation 
of the Brownian motion is only with respect to the variable t (see p. 179). 
That is, we only change the time scale. 


From the formulas in (4.132), we calculate 


~at 


2 
m(az;t) = xz=-—ar and v(2;t)= (e72) e 2a =o” (4.133) 


e` at 


Definition 4.2.6. The stochastic process {U(t),t > 0} whose infinitesimal 
parameters are given by my(u;t) = —au, for allt > 0, where a > 0, and 
vy (u;t) = o° is called an Ornstein—Uhlenbeck process. 


Remarks. i) We see that the Wiener process can be considered as the particular 
case of the O.-U. process obtained by taking the limit as a decreases to zero. 
Conversely, if {U(t),t > 0} is an O.-U. process and if we set B(O) = 0 and 


og? \ 1? 1 2at 
= | — — — for t > 0 4.134 
B(t) (5) U È In ( -3 Ji or ( ) 


then {B(t),t > 0} is a standard Brownian motion. 


ii) We deduce from the definition, given in (4.130), of an O.-U. process in 
terms of {B(t),t > 0} that {U(t),t > 0} is a Markovian process. It is also a 
diffusion process. 


iii) Note that the initial value of an O.-U. process, as defined above, is a 
random variable, since 


U(0) = e°B (<=) =B (=) ~N (0. Z) (4.135) 


We can arrange things so that U (0) = uo, a constant (see Ex. 4.2.3). 





We calculate the mean and the variance of the O.—U. process from (4.130). 
We have 
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g2e22t 
E\U(t)| = £ a (E) = 0 (4.136) 
20 
and 
2 2at 2 2at 2 
VU (t)} = ey B (2 = J! =e? tli Z tor t > 0 (4.137) 
2Q 2a 20 


Next, using the formula Cov|B(s), B(t)] = min{s,¢}, we calculate 


2 ,,2at 2 ,2a(t+s) 
Cov[U (t), U (t + s)] =e% e7% Cov B - ) B ae) 
2a 2 





g2e22t gees 


2a 2a 
Given that the mean of the process is a constant and that its autocovariance 
function Cy(t,t + s) does not depend on t, we can assert that the O.-U. 
process is a wide-sense stationary process (see p. 53). Moreover, since it is a 
Gaussian process, we can state the following proposition. 


—2at ,—as 





=e for s,t > 0 (4.138) 


Proposition 4.2.8. The Ornstein—Uhlenbeck process is a strict-sense station- 
ary process. 


Contrary to the Brownian motion, the increments of the O.-U. process 
are not independent. Indeed, we deduce from the formula (4.138) [and from 
(4.136)] that 


Cov{U(t + s) — U(t), U(t) - U(0) 


= E {U(t + s) — U(t)}{U(t) — U(0)}] -0 x 0 
= E|U(t + s)U(t)] — E[U (t + s)U(0)] — E[U*(t)] + E[U()U(0)] 


og? 


ae Ga — e-otts) 14 ee) 40 (4.139) 

20% 

(if s > 0) so that we can assert that the random variables U (t + s) — U (t) and 
U(t) — U(0) are not independent for all s,t > 0. 

However, as the O.-U. process is (strict-sense) stationary, its increments 
are stationary. To check this assertion, we use the fact that the O.-U. process 
is Gaussian, which implies that the random variable U(t) — U(s), where 0 < 
s < t, has a Gaussian distribution, with zero mean, and whose variance is 
given by 


V[U(t) — U(s)] = VIU()] + VIU(s)] — 2 Cov[U(t), U(s)} 


o? o2e—alt—s) 
= 2 (5 = ae (4.140) 


Now, this variance is identical to that of the variable U(t +7) —- U(s +7), 
which also has a Gaussian distribution with zero mean, from which we can 
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conclude that the random variables U(t) — U(s) and U(t +7) — U(s +7) are 
identically distributed for all r > 0. 


Let us now return to the problem of modeling the displacement of a par- 
ticle, for which we proposed to use a Brownian motion. If we suppose that 
U(t) is the velocity of the particle, then we may write that its position X(t) 
at time t is given by 


X(t) = X(0) + [ U(s) ds (4.141) 


We can therefore assert that X(t) —X (0) has a Gaussian distribution (because 
the O.-U. process is Gaussian), with mean [see (4.136)] 


E[X(t) — X(0)] = E J “U(s) ds = i " ElU(s)] ds = / Ods=0 (4.142) 
and variance [see (4.138)] 


V[X(t) - X(0)] = El[(X(t) - X(0))?] - 0? 


-E| f vo) ds [ u) dr 


t pt 
= f / E{U(s)U(r)] ds dr 
o Jo 
t pt 2 
=f [tester doar 
0 JO 2a 
po pt 


T t 
2 ‘| eal(s—T) as+ | erlt—s) ds} dT 
2a Jo 0 f 


2 pt 
= mal f2 —e@ OT — ett) | dr 
a” Jo 
g? 
DE (at-1+e7%) (4.143) 


From the series expansion of e7%:; 
1 1 
e~ =1-—-at+ 5 (at)” — (at) +... (4.144) 
we may write that 
2 
Z4? if t is small 
Q 
V[X(t) — X(0)] ~ (4.145) 
2 
Zt if t is large 
Q 


Thus, the variance of the integral of the O.-U. process tends to that of a 
Brownian motion when ¢ tends to infinity. However, for small values of t, the 
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variance of the integral is proportional to the square of the time t elapsed 
since the initial time instant. Now, this result is more realistic than assuming 
that the variance is always proportional to t. Indeed, let X ~ N(0,0%). By 
symmetry, we may write that 


E||X]] 2 fo T —z* /(20%) d 
0 V2TOX 


1 = 2 
a xe Cok) = ox (4.146) 
X 0 


Since the standard deviation of X (t)—X (0) is proportional to t for small values 
of t, we deduce from (4.146) that the order of magnitude of the velocity of the 
particle in a small interval of length 6, from the initial time, is a constant: 


v2 (a/V20)5 6 gn sso (4.147) 


m 0 Jra 


whereas the order of magnitude of this velocity tends to infinity in the case 
of the Brownian motion (see p. 176). 


= —2 


The Kolmogorov forward equation corresponding to the O.-U. process is 
the following: 


Op ð a? Op 
pias Egy te SER 4.148 
a Cau e C 


The solution that satisfies the initial condition 


lim p(u, uo; t, to) = d(u — uo) (4.149) 
0 
is 


(u= nov)? } (4.150) 


1 
, ug; t, to) = —— E 
pu i 0) toon exp l 20y 


for ug, u € R and t > tọ > 0, where 


—a (t—to) 


2 
LOU := Uo € and ogy = Z (1 — e722 (t—to)) (4.151) 


2q 
That is, U(t) | {U (tọ) = uo} ~ N(uou, cu) 

We deduce from what precedes that the stochastic process defined from 
a standard Brownian motion in (4.130) is actually the stationary version of 
the Ornstein-Uhlenbeck process, obtained by taking the limit as ¢ tends to 
infinity of the solution above. Indeed, we have 


2 
i = im gy = — 4.152 
a Pou ot: nd Atte 7OU 2a ( ) 
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as in (4.136) and (4.137). 


Remark. The solution (4.150) enables us to state that the O.-U. process is 
time-homogeneous. However, this version of the process is not even wide-sense 
stationary, since its mean depends on t. 


Example 4.2.3. For the initial value of the O.-U. process {U(t),t > 0} to be 
deterministic, we can set, when a = 1 and g? = 2, 


U(t) = uo + e™*B(e”) — B(1) 
We then have 
E[U(t)]| = uo + e *E[B(e*)] — E[B(1)] = uo 
and, for all t > 0, 


V([U(t)] = Vle* B(e”*) — B(1)] 
e*V[B(e™)] + V[B(1)] — 2e-*Cov[B(e™), B(1)] 
aa BOO ptt 4.4 >, =t min{e”", 1} 


=2 — 2e™ x 1 =2(1 —-e™*) 


I! 


4.2.6 The Bessel process 


Let {B,(t),t > 0}, for k = 1,...,n, be independent standard Brownian 
motions. We set 


X(t) = B?(t) +... + B2(t) Vt>0 (4.153) 


The random variable X(t) can be interpreted as the square of the distance 
from the origin of a standard Brownian motion in n dimensions. The following 
proposition can be shown. 


Proposition 4.2.9. The stochastic process {X(t),t > 0} is a diffusion process 
whose infinitesimal parameters are given by 


m(a;t)=n and v(a;t)=4x forx>0 andt>0 (4.154) 


Remark. Note that the infinitesimal parameters of the process do not depend 
on the variable t. 


We now define the process {Y(t),t > 0} by 
Y(t) = g[X(t)] = X(t) fort > 0 (4.155) 


Since the transformation g(x) = xt? is strictly increasing, and its second 
derivative g(x) = —4273/? exists and is continuous for z € (0,00), we can 


4.2 Diffusion processes 205 


use the formulas in (4.35) to calculate the infinitesimal parameters of the 
diffusion process {Y (t),t > 0}: 


DE ae de {1 
LA) Qx1/2 as ~ Ay3/2 


2 = 
2y 2 \ 4y3 Dy 


2 
1 
vy (y) = 42 (<7) =] (4.157) 


and 


It can be shown that this process is indeed Markovian. Moreover, we can 
generalize the definition of the process {Y(t),t > 0} by replacing n by a real 
parameter a > 0. 


Definition 4.2.7. The diffusion process {Y (t),t > 0} whose state space is the 
interval |[0,0o) and whose infinitesimal mean and variance are given, respec- 
tively, by 


and vu(y;t)=1 fort >0andy>0 (4.158) 





Ct nan 
m(y;t) = ay 


is called a Bessel* process of dimension a > 0. 


Remarks. i) The term dimension used for the parameter a comes from the 
interpretation of the process when a = n e€ N. 


ii) When a = 1, the infinitesimal parameters of the process are the same 
as those of the standard Brownian motion. However, we can represent it as 
follows: 


Y(@)=|B@)| fort >0 (4.159) 


where { B(t),t > 0} is a standard Brownian motion. It is as if the origin were 
a reflecting boundary for {B(t),t > 0}. 

The conditional transition density function of the Bessel process satisfies 
the Kolmogorov forward equation 


_ 2 
Op _l-að /p a 1d"p (4.160) 
ot 2 Oy ly 2 Oy? 

We can check that 

FERE ye +y? Yoy 
tt =02=2 il ae a e 4.161 
p(y, Yo; t, to 0) } (2) y exp { J ( t ) ( ) 


4 See p. 114. 
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for yo, y, and t > 0, where 


pes a] (4.162) 
2 
and I,(-) is a modified Bessel function of the first kind (of order v), defined 


by (see p. 375 of Ref. [1]) 


oc 2 k 
I, (z) = (2/2)” 2, aaa (4.163) 


Remark. The quantity v defined in (4.162) is called the indez of the Bessel 
process. 


Definition 4.2.8. The diffusion process {X(t),t > 0} defined by 
X(t)=Y7(t) fort >0 (4.164) 


is called a squared Bessel process of dimension a > 0. Its infinitesimal 
parameters are given by 


m(z;t) =a and v(z;t)=4r fort >0 andx>0 (4.165) 


Remarks. i) We deduce from the representation of the process {X(t),t > 0} 
given in (4.153) that if {X (t),t > 0} and {Xo(t),t > 0} are two independent 
squared Bessel processes, of dimensions a; = nı and a2 = nz, respectively, 
then the process {S(t),t > 0}, where 


S(t) := Xi (t) + Xe(t) fort >0 (4.166) 


is also a squared Bessel process, of dimension a := nı + ng. Actually, this 
result is valid for all a; > 0 and az > 0, and not only when a; and a are 
integers. 


ii) The function p(z, z9;t,t9 = 0) of the squared Bessel process of dimension 
Q is given by 


v/{2 — 
p(z, Xo; t, tp = 0) = (= exp f- T = Í; ( =) (4.167) 


0 2t t 








for Zo, x, and t > 0, where v = (a/2) — 1. 


A diffusion process used in financial mathematics to model the variations 
of interest rates, and which can be expressed in terms of a squared Bessel 
process, is named the Cox—Ingersoll—Ross°- (CIR) process (see Ref. [4}). 
We set 


5 John C. Cox and Stephen A. Ross are professors at the MIT Sloan School of 
Management. Jonathan E. Ingersoll, Jr. is a professor at the Yale School of Man- 
agement. | 
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g? 


R(t) =e" X ( T 


C= D) for t > 0 (4.168) 


where b € R and o > 0 are constants, and {X(t),t > 0} is a squared Bessel 
process of dimension a = 4a/a* (with a > 0). We find that the infinitesimal 
parameters of the process {R(t),t > 0} are given by 


m(r;t)=a—br and v(r;t)=o’r fort >Oandr>0 (4.169) 


From the formula (4.167), we find that 


+ re Tare 
p(T, ro; t, to = 0) = : (=) exp { yaf 0 
0 





2k(t) 2k(t) k(t) 
(4.170) 
for ro, r, and t > 0, where 
2 
ERT 2a 
a = — := — — 1 4.171 
k(t) TAG 1) and v -2 ( ) 


4.3 White noise 


In Section 2.4, we defined the stochastic process called white noise as being 
a process (that we now denote by) {X(t),t > 0} with zero mean and whose 
autocovariance function is of the form 


Cx (ty, t2) = q(t, )d(te = tı) (4.172) 


where q(tı) is a positive function and 6(-) is the Dirac delta function (see 
p. 63). 

When q(t1) = o°, the function Cx (t1, t2) is the second mixed derivative of 
the autocovariance function Cw (t1, t2) = o? min{t;,¢2} of a Brownian motion 
(without drift and) with diffusion coefficient a7. Indeed, we have 


ð T 0 ifti < te 
Dey CW ata) aa f ift > te (4.173) 
so that 
ee eo to) — 2 Pilt = tz) = o75(ty — t2) = o75(to ai ti) (4.174) 
otito ? oti 


where u(-) is the Heaviside function (see p. 11). 


Definition 4.3.1. The (generalized) stochastic process {X (t), t > 0} with zero 
mean and autocovariance function 


Cx (t1,t2) = 07d (te — t1) (4.175) 


is called a Gaussian white noise (or white Gaussian noise). 
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Remarks. i) Since E[X(t)| = 0, we also have 


Rx (ti, te) = = g? d(t2 = tı) (4.176) 


The second mixed derivative DLO a Ry (t,t) is called the generalized mean- 
square derivative of the stochastic process {Y(t),t > 0}. i say that the 
process {.X(t),t > 0} having the autocorrelation function TN 55 Ry (tı, t2) is 
a generalized stochastic process. 

ii) A white noise (Gaussian or not) is such that the random variables X(t1) 
and X(t2) are uncorrelated if ti # t2. If the variables X(t) and X(t2) are 
independent, the expression strict white noise is used to designate the corre- 
sponding process. 


Now, we mentioned that the Brownian motion is nowhere differentiable. 
Consider, however, the process {X(t),t > 0} defined (symbolically) by 


X(t) = lim a al) (4.177) 


€|0 € 


where € > 0 is a constant and {W (t),t > 0} is a Brownian motion with 
coefficients 4 = 0 and o? > 0. Assume that we can interchange the limit and 
the mathematical expectation. Then 


E[X(t)] = lim E eee) =lim0 = 0 (4.178) 


el 
so that 


Cott) = tig [( Weta Wit)) (Wierd = Wie) 


€ € 


= = lim 3 i {E [W (ty + e)W (te + €)] -E [W (ti + EW (t2) 
— E[W(t,)W(te + ©€)] + E [W (t1) W (te) } (4.179) 


Suppose first that tı < t2. Then, for an € small enough, we will have that 
tı +€ < tg. It follows that 


Cx (ty, te) = = lim 7 =A -+ €) = (ti + €) — tı + i= = lim = — Se 0 (4. 180) 
When ti = t2, we obtain 


Cx (ti, t2) = = lim = =r T + €) =< ti — t; + ti} 


2 g? 


= lim z e = lim — = œ (4.181) 
ejO € cel € 
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We can therefore write that 
Cx (t1, te) = 075(te — t1) (4.182) 


Thus, we interpret a Gaussian white noise as being, in a certain way, the 
derivative of a Brownian motion. From now on, we will denote the Gaussian 
white noise by {dW(t),t > 0}. We also have that dW(t) = W(t) dt. In 
engineering, the notation dW (t)/dt = e(t) is often used. 


Even though the Brownian motion is not differentiable, we can use the 
notion of a generalized derivative, which is defined as follows. 


Definition 4.3.2. Let f be a function whose derivative exists and is continu- 
ous in the interval [0,t]. The generalized derivative of the function W(t), 
where {W(t),t > 0} is a Brownian motion, is given by 


/ f(s)W'(s) ds = f(W (t) - | W (s) f(s) ds (4.183) 
0 0 


Remarks. i) This definition of the generalized derivative of W(t) is actually a 
more rigorous way of defining a Gaussian white noise {dW(t),t > 0}. 


ii) For a deterministic function g, its generalized derivative is defined by 


OO OO 
f Eo as =— | aee) ds (4.184) 
0 0 
where we assume that f(t) = 0 for all t é [a,b], with a > 0 and b < œ. 


The formula (4.183) leads us to the notion of a stochastic integral. 


Definition 4.3.3. Let f be a function whose derivative exists and is contin- 
uous in the interval [a,b], where a > 0, and let {W(t),t > 0} be a Wiener 


process. We define the stochastic integral i f(t) dW(t) by 
b b 
J tawe=1oww-saw@-[wHae — (4.185) 


Remarks. i) We can also define a stochastic integral as follows: 


b n 
[ saw = lim S F(te_1)[W (ts) — W(ti-1)) (4-186) 
fe. i=l 
max {ti —t;-1} 10 


where a = tọ < tı <...<t, =b is a partition of [a,b]. 


ii) The definition of ie f(t) dW (t) follows from the formula (4.177) as well, 
by setting 
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7 " F(t) aW(t) = lim / oo (=e To) dt (4.187) 


Indeed, using the formula 


— t+e 
W(t+e)- W(t) d ( 1 W(s) is) (4.188) 
€ dt \e ji 
and integrating by parts, we obtain 


t+e b 


J O awg) = lim f HOF W(s) ds| 


2 [ f(t) G Ws) ds) at} (4.189) 


t 


Finally, since W(t) is a continuous function, we have (making use of 
l’Hospital’s® rule) 


t+e tte 


lim — : W(s) ds = ae : 


= lim W = W(t) (4.190 
el0 € 0 ae W(s) ds = lim (t +e) (t) (4.190) 


from which we retrieve the formula (4.185). 


Properties. i) From the formula (4.185), we can assert that a stochastic 
integral has a Gaussian distribution, because it is a linear combination of 
Gaussian random variables. 


ii) We have 





elf W(t) af D| =0 
(4.191) 


b 
E / f(t) WO = SOE) - f(a) E[W(a)| — 





where we assumed that we can interchange the mathematical expectation and 
the integral. 


iii) To calculate the variance of a stochastic integral, we can use the formula 
(4.186) and the fact that the increments of the Brownian motion are indepen- 
dent and stationary. We have 


f?(ti-1)V[W (ti) — W(ti-1)] 


Me: 


v |E re i) — W(t, D = 


e, 
li 
n 


f? (ti-1)V[W (ti — ti-1)] 


Ms 


1 


e, 
I 


® Guillaume Francois Antoine de Hospital, 1661-1704, was born and died in 
France. He published the first textbook on differential calculus, in which the 
rule that bears his name can be found. 
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= 5 F’ (ti-1)0°(ti — ti-1) (4.192) 


i=1 


It follows (interchanging the limit and the calculation of the variance) that 





b n 
V / F(t) dW (t) = lim S P tiio (ti = ti—1) 
s n — CO i=l 
max {ti —t-1} | 0 
=g’ / Pty dt (4.193) 


iv) We can generalize the preceding formula as follows: 


0 ifa<b<c<d 
E 





b d 
/ f(t) aW (t) / g(t) awo) = b 
a c o | f(t)g(t) dtifa=c<b<d 


(4.194) 


where g is a function whose derivative exists and is continuous in the interval 
[c,d]. 


Let {X(t),t > 0} be a continuous-time and continuous-state stochastic 
process whose infinitesimal parameters are m(x;t) and v(x;t). This process 
can be represented in the following way (see Ref. [16], for instance): 


X(t) = X(0) +f m[X (s); s] ds +f v/2/X(s); s] dB(s) (4.195) 


where {B(t),t > 0} is a standard Brownian motion. It follows that X(t) is a 
solution of the stochastic differential equation 


dX(t) = m[X(t);t] dt + v1/?[X(t);t] dB(t) (4.196) 
Under the condition X(t) = x, the equation above becomes 


dX(t) = m(x;t) dt + v'/?(x;t) dB(t) (4.197) 
<= > X(t) =m(a;t) +v? (zt) Bt) (4.198) 
(with the notation X(t) = £ X(t). 


We can consider stochastic differential equations in n dimensions. A useful 
result is given in the following proposition. 


Proposition 4.3.1. Let {X(t),t > 0} be an n-dimensional stochastic process 
defined by 
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dX(t) = (AX(t) + a) dt + N+? dB(t) (4.199) 


where {B(t),t > 0} is an n-dimensional standard Brownian motion, A is a 
square matriz of order n, a is an n-dimensional vector, and N‘/? is a positive 
definite square matrix of order n. Then, given that X(to) = x, we may write 
that 


X(t) ~ N(m(t), K(t)) fort > to (4.200) 
where 
m(t) := (t) (x + [ @ (ula du) (4.201) 
and 


K(t) := (t) (f D- LHu)N|Ð u) du) D' (t) (4.202) 


to 


where the symbol prime denotes the transpose of the matriz, and the function 
P(t) is given by 


= (t — to)” 
Ober) ay ar ae (4.203) 
n=0 


Remarks. i) A matrix M = (mi j); j=1,... n 1S positive definite if 
yf ~T Agere ; 


n n 
c'Mc = D >». CicCjMi j > 0 (4.204) 
i=1 j=1 
for any vector c’ := (cj,... ,Cn) that is not the null vector (0,... ,0). 


ii) When n = 1, the formulas for the mean and the variance become 
x + a(t — to) if A=0 
G\ WA(t-to) _ & if AHO 
(2+ a) : er 
and 
N(t — to) if A=0 
K(t) = (4.206) 
N (2A(t-to) 
a o) 0 
24 (e 1) if AF 


iii) We can generalize the proposition to the case where A = A(t), a = a(t), 
and N!/? = N1/2(¢), The function &(t) is then obtained by solving the matrix 
differential equation 


Za) = A(t)S(t) (4.207) 


with the initial condition &(t9) = 1, (the identity matrix of order n). 
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Ezample 4.3.1. The Brownian motion {X(t),t > 0} with drift coefficient 4 # 
0 and diffusion coefficient a? (> 0) is defined by the stochastic differential 
equation 

dX (t) = pdt + odB(t) 


According to the second remark above, the random variable X(t), given that 
X (0) = z, has a Gaussian distribution with parameters 


xæ = m(t)=a2+ pt and ox) = K(t)=0°t 


which does correspond to the results mentioned in Subsection 4.2.2. 


In the case of the Ornstein-Uhlenbeck process, {U (t), t > 0}, we have the 
following stochastic differential equation: 


dU(t) = ~a U(t) dt +0 dB(t) 


so that A = —a, a = 0, and N!/2 = ø. It follows that the random variable 
Y(t) := U(t) | {U(to) = uo} has a Gaussian distribution whose parameters 
are (see p. 203) 


2 
Ly (t) = Uo e72lt—to) and F(t) = = Cae — 1) 


Example 4.3.2. Consider now the two-dimensional diffusion process X(t) = 
(Z(t), Y(t)) defined by the system of equations 


dZ(t) = Y(t) dt 
dY (t) = u dt + odB(t) 


That is, {Y (t), t > 0} is a Brownian motion with drift coefficient u (€ R) and 
diffusion coefficient o° (> 0), and {Z(t),t > 0} is its integral. Suppose that 
(Z(0), Y(0)) = (z, y). We may write that 


_ {01 _ 10 1/2 {00 
a= loo Si and N =lo0| 


Let us first calculate the function (t). We have 
ə |O1];;01; |00 
A k 0} ;/00] |00 


Lt 
o() =Ta+at= [51] 


It follows that 


oe Pea | ane Steel 
w= (29 osna 
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[ow a du = a Fa du = Ea 
0 0 H pt 


which implies that 


TERIGA) 


Next, we have 
_ {00 
~ 1007 


[roor EREN 
=f Meee | ae 


| | es 07 [2 
~ | -o7t?2/2 ot 


We then have 


z+ yt+ ppt? 
y + pt 


Z 


and 


from which we calculate 


xy = [5] [ a [10] [ee oe? 


Note that these results agree with those mentioned in the case of the three- 
dimensional process (Y (t), Z(t), D(t)) (see p. 196). 


4.4 First-passage problems 


Let Ty be the random variable that designates the time needed for a standard 
Brownian motion to go from B(0) = 0 to B(t) = d # 0. Symbolically, we 
write 


Ta := min{t > 0: B(t) =d} (4.208) 


Remark. To be more rigorous, we should write that Tg is the infimum (rather 
than the minimum) of the positive values of t for which B(t) = d. Indeed, in 
the case of the standard Brownian motion, we can show [see Eq. (4.218)] that 
the probability that the process will eventually hit the boundary at d is equal 
to 1, for any d € R. However, in other cases, it is not certain that the process 
will hit the boundary at d. If the stochastic process does not hit the boundary, 
the set whose minimum is to be found is then empty and this minimum does 
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not exist. On the other hand, the infimum of the (empty) set in question is, 
by definition, equal to infinity. 


Suppose first that d is a positive constant. We can obtain the distribution 
of the random variable Tg by using the fact that the distribution of B(t) is 
known, and by conditioning on the possible values of Ty. We have 


P[B(t) > d] = P[B(t) > d | Ty < t]P\Ty < t) + PIB(t) > d | Ta > t]P|Ta > J 
(4.209 


We deduce from the continuity of the process {B(t),t > 0} that 
P|B(t) > d| Ta >t] =0 (4.210) 
Moreover, if Ty < t, then there exists a tp € (0, t) such that B(tg) = d. Since 
Bit) | {B(to) = d} ~ N(d,t — to) fort > to (4.211) 


we may write (by symmetry) that 


PIB(t) > d| Ti < t] = (4.212) 


Finally, P[B(t) > d | Ty = t| = 1. However, as Ty is a continuous random 
variable (which implies that P[Ty = t] = 0), we may conclude that 


PIT, < t}=2 i = P(d/ v2) | where (x) := PIN(0,1) <æ] (4.213) 


Remarks. i) The preceding formula is called the reflection principle for the 
Brownian motion. 


ii) We also deduce from the continuity of the Wiener process that 
P max BC) >d (> 0) Siig 2 i — P(d/ vE) (4.214) 


That is, the probability that the process takes on a value greater than or equal 
to d > 0 in the interval [0, t} is the same as the probability that it reaches the 
boundary at d not later than at time t. 


iii) By symmetry, Ta and T_g are identically distributed random variables. It 
follows that 


PIT, < t] =2 i = 2(\d\/v*)| Vd40 (4.215) 
iv) The density function of Ty is obtained by differentiating the function above: 


\d|/Vt 1 F 
2i;1- ———e* [2 dz 
Ce ager) 


fr,(t) = 5 
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an d) (__ldi 
~ “Van PU tf 2w 


__id d” 
ae: exP 1 — 55 for t > 0 (4.216) 








This density is a particular case of the inverse Gaussian or Wald’ distribution. 


Remark. The density function of a random variable X having an inverse Gaus- 
sian distribution with parameters u > 0 and A > 0 is given by 





A N(x — 2)? 
Fx (osm) = a exp { AZT Vai>0 (4.217) 


The distribution of T4 above is thus, in fact, obtained by setting À = d? and 
by taking the limit as the parameter u tends to infinity. 


Note that from (4.215), we deduce that 
| 1 
Pilg as Jim P(Ta < t] = 2[{1 — e02 (1 — 5) =1 (4.218) 


Therefore, the standard Brownian motion is certain to eventually reach any 
real value d. However, the formula (4.216) implies that | 


EIT, = n |d| Ë dt = œ (4.219) 
HT Jo Ven PAS | 
because 

1 d? 1 


1/2 


That is, the function that we integrate behaves like t™*/ as t tends to infinity. 


Given that 


a 
/ t-1/2 dt=o0 VYto>0 (4.221) 
t 


0 
we must conclude that the mathematical expectation of the random variable 
Ta is indeed infinite. 
We also deduce from the formula (4.215) that 


Pid?T, < t] =2 i — (1/ ire) = 2 E —@ (\al/ve) | (4.222) 


T Abraham Wald, 1902-1950, was born in Kolozsvár, Hungary (now Cluj, in Roma- 
nia), and died (in a plane crash) in India. He first worked on geometry and then 
on econometrics. His main contributions were, however, to statistics, notably to 
sequential analysis. 
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Thus, the random variables Ty and Ty := d?T, are identically distributed. 


Finally, let X := 1/B?(1). Since B(1) has a Gaussian N(0, 1) distribution, 
we calculate, for x > 0, 


PX <a] = P[B*(1) > 1/2] = 1 — P[-1/Vz < B(1) < 1/vz | 
= 1- (2 8 (1/v2) -1) =2 (1 -8 (1/v7)) (4.223) 


from which we can assert that T; and X are also identically distributed ran- 
dom variables. 


Remark. The square of a standard Gaussian random variable Z has a chi- 
square distribution with 1 degree of freedom, which is also a particular gamma 
distribution: Z? ~ G(1/2, 1/2). The random variable T; is thus the reciprocal 
of a variable having a gamma distribution. 


A much more general technique to obtain the distribution of a first-passage 
time T for an arbitrary diffusion process consists in trying to solve the Kol- 
mogorov backward equation satisfied by the density function of this random 
variable, under the appropriate conditions. 

Let {X (t), t > 0} be a time-homogeneous diffusion process whose infinites- 
imal parameters are m(x) and u(x). Its conditional transition density function, 
p(x, £o; t, to), satisfies 


Op dp 1 0p 
ies — = 4.224 
9 (0) Ox? i ( 


If to = 0, we can use the fact that the process is time-homogeneous, so that 


ð ð ð 
— * == — $ ° — E oain “$ S t 4.225 
zg? £0; t, to) ge, er tort to) P(T x0; 0) ( ) 


to write that 


1 A? Op Op 
Sal i Se 4.226 
5 (zo) Ox ae m{Xo) 0X6 Ot ( ) 


Let p(t; xo) be the probability density function of 
Tea (= Te,a(xo)) := min{t > 0: X(t) ¢ (c,d) | X(0) = zo € |e, d]} (4.227) 
That is, 
p(t; zo) dt := P[T oq € (t,t + dt} | X(0) = zo € [e, d]} (4.228) 


We find that the function p(t; zo) satisfies the partial differential equation 
(4.226). Moreover, since the coefficients m(xg) and v(x) do not depend on 
t, we can take the Laplace transform of the equation (with respect to t) to 
reduce it to an ordinary differential equation. Let 
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OO 
ee | e-** p(t: ary) at (4.229) 
Q 


where a is a real positive constant. 


Remark. The Laplace transform of the density function of the continuous and 
nonnegative random variable X is the moment-generating function Myx(-—a) 
of this random variable (see p. 19). 


We can check that the function L(xo; œ) satisfies the following differential 
equation: 
1 ðL OL 
-= — — =al 4,230 
zeo) + mao) 5 =a (4.230) 


This equation is valid for c < x9 < d. Indeed, we have 


j MEKA (t; xo) dt = e7% p(t; zolo + a | e7% p(t; £o) dt 
0 ot 0 
=0+aL=al (4.231) 


because p(0; xo) = 0 if zo # cor d. Since Tea = 0 if zo = c or d, the boundary 
conditions are 


L(apj;a) = E [e ora] X(0)=zoọ] =1 ifro=cord (4.232) 


Once we have found the function L(x9;a), we must invert the Laplace 
transform to obtain the density function of Tea- 


Example 4.4.1. Let {X(t),t > 0} be a Wiener process whose drift and dif- 
fusion coefficients are u and o%, respectively. We must solve the following 
differential equation: 
o OL ðL 
za aa 


The general solution of this equation is given by 


aL 


L(z0;@) = ce”? (0) 4 ege™r2() (4.233) 


where c; and cz are constants, and 
1 1 
ry(Q) := -z (-u — y u? + 2007) and ro(a) := z (-n +y u? + 2a?) 
The solution that satisfies the conditions L(c;a) = L(d; a) = 1 is 


(ed rala) _ ef r2(a)) eror1(a) Ez (e° rile) _ ef r1(a)) etor2(a) 


L(x0; a) = ecTi (a)+d ro{a) — ef r2(a)+d rı(a) 


(4.234) 


for € < £o < d. 
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Inverting the Laplace transform above is not easy. However, in this type 
of problem, we must often content ourselves with explicitly calculating the 
moment-generating function of the first-passage time considered. 


Suppose, to simplify, that there is a single boundary, at d (which is tanta- 
mount to taking c = —oo) and that u = 0. Since, from Eq. (4.232), L(xo; a) 
must be in the interval [0,1], for any x79 < d, we must discard the solution 
with rı (a) in (4.233). Indeed, with u = 0, this solution is 


cje V ato T es Ci X OO as zro — -œ 


We must therefore choose the constant cı = 0. We then have 


L(z0; @) = cgeY 2020/0 


and the condition L(d; a) = 1 implies that 


L(x9; a2) = eV?*20-4)/0 Vag <d (4.235) 


Remark. We obtain the same result by taking the limit as c decreases to — oo 
in the formula (4.234) (with u = 0). 


We can check that the inverse Laplace transform of the function L in Eq. 
(4.235) is given by (see the inverse Gaussian distribution, p. 216) 


fr,(t) = C exp {Coal for t > 0 
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The density function of the random variable Ty in the general case where 
ue Ris | 


= (d— zo) (d — zo — pt)? 
ÍT, (t) => Jno exp Da for t > 0 (4,236) 


and the function L becomes 


DL (x9; a) = exp Ao a — (p + 2ag?)"/?] ) for zo <d (4.237) 


After having obtained the moment-generating function of the random vari- 
able Tq, we can calculate the probability that this variable is finite, as follows: 


Pilg< oo) = lim L(x; a) (4.238) 


Example 4.4.2. In the preceding example, with a single boundary and p = 0, 
we obtain that 


P(Ty < œ] = lim ev 2a(xo-d)/o _ 4 
a 
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which generalizes (in the case when d > 0) the result already mentioned 
regarding the standard Brownian motion. 
However, when u # 0, we deduce from the formula (4.237) that 


a {lO oO 


E 1 if u>O0 
~ | e2e(d-20)/o* if <0 


To obtain the moments of the random variable Ty, we can use the formula 
(see p. 19) 





we” 
ET?) = (-1) aan L (x9; &)| <0 (4.239) 
for n = 1,2,.... Thus, in Example 4.4.1, if c = —co and p > 0, we calculate 


[see the formula (4.237)| 











ð d— zx 
E [Ta] = -57 exp piigi u — (p? + 2a07)1/2| , 
= —L(z0;a) ($ ==) Ir 2007) 446" 
a==0 
d — To 
= —__ 4,240 
7 (4.240) 


Remark. We can also try to solve the ordinary differential equation satisfied 
by the function 


maalto) = E7] for n= 1,2,... (4.241) 


namely, 


1 d? d 
5 (20) 7 a Mn a(20) F m(zo) 7 -Mn,d(o) cae ™Mn—1,a(Xo) (4.242) 
0 


under the boundary condition Mn a(d) = 0. In particular, we have 


1 d? d 
30) a + m(zo) 7 —™m1,a(z0) aml (4.243) 


When there are two boundaries, if we wish to calculate the probability 
that the process will hit the boundary at d before that at c, we can simply 
solve the ordinary differential equation 


1 d? d 
go (To) grga lto) T m(zo) Tp Palto) =0 (4.244) 
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where 
Dito) = PIX (Tato) Sa (4.245) 


This differential equation is obtained from Eq. (4.230) by setting a = 0. 
Indeed, we may write that 


p(t; £0) = pet; xo) + palt; z0) (4.246) 


where pe(t; £o) (respectively, pa(t; £o)) is the density function of the random 
variable T% (resp., T}) denoting the time the process takes to go from xg to c 
(resp., d) without hitting d (resp., c). Let 


Lalz£o; a) = [ e7% palt; £o) dt (4.247) 
0 
We have 
P|X (Te alzo)) = d) = PIT; (xo) < oo] = f palt; xo) dt = Lalxo;0) (4.248) 


Now, the function Lg(xo; a) also satisfies Eq. (4.230). 
The differential equation (4.244) is valid for c < xp < d. The boundary 
conditions are 


pa(d)=1 and pg(c) =0 (4.249) 


Example 4.4.8. If {W(t),t > 0} is a Wiener process with diffusion coefficient 
o*, then we must solve 

o? g 0 

z daa iTo) T 
We find at once that 

Palzo) = €1L0 + Co 


where Co and cı are constants. The boundary conditions imply that 


Tto — ce 
=-———  force<ap<d 
pa(to) = -7 fore < a9 S 
Note that this formula does not depend on a”. Moreover, if c = —d, then we 
have that pg(0) = 1/2, which could have been predicted, by symmetry. 


Until now, we only considered the time a diffusion process takes to reach 
a boundary or either of two boundaries. We can define these boundaries as 
being absorbing. If the boundary at c is reflecting, it can be shown that the 
boundary condition becomes 
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o 
—— L(x; a) = 0 (4.250) 
Oxo Ly=HC 
Finally, we can also try to calculate first-passage time distributions in two 
or more dimensions. However, the problem of solving the appropriate partial 
differential equation is generally very difficult. 


Example 4.4.4. If {B(t),t > 0} is a standard Brownian motion, starting from 
bo, and if {X(t),t > 0} is its integral, we find that the Kolmogorov backward 
equation satisfied by the function 


P\Ta(bo, E€ (t,t+ dt 
p(t; bo, £o) u | al 0 a ( lI 


where 
Ta(bo, zo) := min{t > 0: X(t) = d > 0 | B(0) = bo, X(0) = zo < d} 


is the following: 

18 p a _ a 

20b °° Oxy ðt 
The main difficulty is due to the fact that the process {X (t), t > 0} cannot hit 
the boundary X(t) = d (for the first time) with B(t) < 0. Indeed, we deduce 
from the formula 

j d 
AEE / Bs) ds = X(t) = Bet 

0 
that if B(t) takes on a negative value, then X(t) decreases. Consequently, 
the process {X (t),t > 0} can attain a value d (> 0) greater than zo at time 
Ta(bo, zo) only if B[Ta(bo,20)} > 0, so that the function p is not continuous 
on the boundary. 


4.5 Exercises 


Section 4.1 


Remark. In the following exercises, the process {B(t),t > 0} is always a stan- 
dard Brownian motion. 


Question no. 1 
We define 


X(t) = B(t) fort >0 


(a) Is the stochastic process {X(t),t > 0} a Wiener process? Justify. 
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(b) Is {X(¢),t = 0} (i) wide-sense stationary? (ii) mean ergodic? Justify. 


Question no. 2 
We consider the stochastic process {X (t), t > 0} defined by 


X(t) = B(t)+ B(t?) fort >0 


(a) Calculate the mean of X(t). 

(b) Calculate Cov[X(t), X(t + 7)], for 7 > 0. 

(c) Is the stochastic process {X(t),t > 0} (i) Gaussian? (ii) stationary? (ili) a 
Brownian motion? Justify. 

(d) Calculate the correlation coefficient of B(t) and B(t?), for t > 0. 
Question no. 3 


Calculate the variance of the random variable X := B(t) — 2B(r), for 
QO<t<rT. 


Question no. 4 
Let X(t) := |B(t)|, for t > 0. Is the stochastic process {X(t),t > 0} 
(a) Gaussian? (b) stationary? Justify. 


Question no. 5 
Let {X(t),¢ > 0} be the stochastic process defined by 


X(t)= B(t+1)- B(1) fort >0 


(a) Calculate the autocovariance function of the process {X(t),t > O}. 


(b) Is the process {X(t),t > 0} (i) Gaussian? (ii) a standard Brownian mo- 
tion? (iii) stationary? (iv) mean ergodic? Justify. 


Question no. 6 
Calculate the variance of X := B(4) — 2B(1). 


Question no. 7 
Is the stochastic process {X(t),t > 0} defined by 


X(t)=-B(t) fort >0 


Gaussian? Is it a Brownian motion? Justify. 


Question no. 8 
Let {X(t),¢ > 0} be a Gaussian process such that X(0) = 0, E[X(t)] = ut 
if t > 0, where u Æ 0, and 


Rx(t,t +7) = 22+ yp°t(t+7) fort,r >0 


Is the stochastic process {Y(t),t > 0}, where Y(t) := X(t) — ut, a Brownian 
motion? Justify. 
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Question no. 9 
Is the Wiener process mean ergodic? Justify. 


Question no. 10 
We set i 
ros 7B? (t) for t > 0 


(a) Is the process {Y(t),t > 0} Gaussian? Justify. 
(b) Calculate the mean of Y(t). 

(c) Calculate Cov[Y(s), Y(t)], for 0 <s <t. 
Indication. If X ~ N(0,0%) and Y ~ N(0,0%,), then 


E[X?Y?] = E[X?|ElY?] + 2(£[XyY])’ 


(d) Is {Y (t),t > 0} a wide-sense stationary process? Justify. 


Question no. 11 
Consider the stochastic process {X (t), t > 0} defined by 


where ft] denotes the integer part of t. 

(a) Calculate the mean of X(t). 

(b) Calculate Cov[X (t1), X (t2)], for t2 > tı. Are the random variables X (tı) 
and X(t2) independent? Justify. 

(c) Is the stochastic process {X(t),t > 0} (i) stationary? (ii) a Brownian 
motion? Justify. 


Question no. 12 

At each time unit, the standard Brownian motion {B(t),t > 0} is shifted 
to {B(n)}, where [ ] denotes the integer part. Let Xn be the position at time 
n, for n = 0,1,2,.... Then the process {X,,n = 0,1,...} is a (discrete-time) 
Markov chain. Calculate (a) p;,;, for i,j > 0 and (b) P[X; = 0, X2 = 0}. 


Question no. 13 


We define 
X(t) = Biln(t+1)) fort >0 


(a) Is the stochastic process {X(t),t > 0} a Brownian motion? Justify. 
(b) Calculate E[X?(t) | X(t) > 0]. 


Question no. 14 
Let {X(t),t > 0} be the stochastic process defined by 


Vti>0 


xq) = Bets 9 — B(t) 
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where € is a positive constant. 

(a) Calculate Cx (t,t + s), for s,t > 0. 

(b) Is the process {X(t),t > 0} (i) Gaussian? (ii) stationary? (iii) a Brownian 
motion? (iv) mean ergodic? Justify. 

Question no. 15 


Let X1, Xo, ... be independent and identically distributed random vari- 
ables such that px, (x) = 1/2 if x = —1 or 1. We define Yo = 0 and 


Y= for WL, 2h a5 
k=1 


Then the stochastic process {Y,,n = 0,1,...} is a Markov chain (see p. 85). 
We propose to use a standard Brownian motion, {B(t),t > 0}, to approx- 

imate the stochastic process {Y,,n = 0,1,...}. Compare the exact value of 

P[Y30 = 0] to P{-1 < B(30) < 1]. 

Indication. We have that PIN(0,1) < 0.18] ~ 0.5714. 


Section 4.2 


Question no. 16 
Let {U (t), t > 0} be an Ornstein-Uhlenbeck process defined by 


U(t) = e~*B(e**) 


(a) What is the distribution of U(1) + U(2)? 
(b) We set 


Vi)= [ U(s) ds 


(i) Calculate the mean and the variance of V(t). 
(ii) Is the process {V(t),¢ > 0} Gaussian? Justify. 


Question no. 17 

Suppose that {X (t), t > 0} isa Wiener process with drift coefficient M and 
diffusion coefficient o? = 1, where M is a random variable having a uniform 
distribution on the interval (0, 1]. 


(a) Calculate E[X(t)} and Cov[X(s), X(t)], for s,t > 0. 
(b) Is the process wide-sense stationary? Justify. 
Question no. 18 

Let {Y(t),t > 0} be a geometric Brownian motion. 


(a) Show that the density function of the random variable Y(t) is given by 
the formula (4.52). 


(b) Is the process {Y (t), t > 0} stationary? Justify. 
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Question no. 19 


Let {X(t),t > 0} be a Wiener process with drift coefficient u > 0 and 
diffusion coefficient o° = 1. 


(a) (i) Calculate, as explicitly as possible, E[X(t) | X(t) > 0] in terms of 
Q(x) := P[N(O, 1) > z]. 
(ii) Calculate ELX(t)X (t + s)|, for s,t > 0. 
(b) Let Z(0) = 0 and 
Z(t) eae, fort > 0 
pt 


Is the stochastic process {Z(t),¢ > 0} a Brownian motion? Justify. 


Section 4.3 


Question no. 20 
(a) Calculate the covariance of X and Y, where 


1 1 
X =| tdB(t) and Y = | t? dB(t) 
-1 —1 


(b) Are the random variables X and Y independent? Justify. 


Question no. 21 
We define the stochastic process {Y (t), t > 0} by 


Y= J» dB(s) fort>0 


(a) Calculate the autocovariance function of the process {Y (t), t > 0}. 

(b) Is the process {Y (t),t > 0} Gaussian? Justify. 

(c) Let {Z(t),t > 0} be the stochastic process defined by Z(t) = Y? (t). 
Calculate its mean. 


Question no. 22 
Calculate the mean and the autocovariance function of the stochastic pro- 
cess {Y (t),t > 0} defined by 


t 
Y= et f e “dW(s) fort >0 
0 


where c is a real constant. 
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Section 4.4 


Question no. 23 
Let {W(t),t > 0} be a Brownian motion with infinitesimal parameters 
u = 0 and o° (> 0). We set 


Y(t)= W(t?) fort >0 


(a) Is the stochastic process {Y(t),t > 0} a Brownian motion? Justify. 
(b) Calculate Cov[Y(s), Y(t), forO<s <t. 
(c) Let Ta := min{t > 0: Y(t) = d > 0}. Calculate the probability density 
' function of the random variable Ty. 
Question no. 24 

Let {Y(t),t > 0} be the stochastic process defined in Question no. 21. 
(a) Calculate the distribution of Y(1) + Y(2). 
(b) Let Tı := min{t > 0: Y(t) = 1}. Calculate the probability density function 
of the random variable 7}. 


Question no. 25 
We consider the process {X(t),t > 1} defined by 


X(t) =e“ B(eM*) fort >1 


Suppose that B(e) = 0, so that X(1) = 0. 
(a) Calculate EX (t)] and Cov[ X(t), X(t + s)], fort > 1,8 > 0. 


(b) Is the stochastic process {X(t),t > 1} (i) Gaussian? (ii) an Ornstein- 
Uhlenbeck process? (iii) stationary? Justify. 


(c) Let Ta := min{t > 1: X(t) = d > 0}. Calculate fr, (t). 


Question no. 26 
Let {Z(t),0 <t <1} be a Brownian bridge. We define 


t 
v=] Z(r) dr forO<t<1 
0 


(a) Calculate E[Y (t)] and Cov[Y (t), Y(t+s)], forO<t<1,s2>0,s+t<1. 
(b) Is the stochastic process {Y(t),0 < t < 1} (i) Gaussian? (ii) stationary? 
(iii) a Brownian bridge? Justify. i 

(c) Calculate approximately, if d > 0 is small, the probability that the process 
{Y (t),0 <t < 1} will reach d in the interval (0,1). 


Question no. 27 
Let Te be the first-passage time to the origin for a standard Brownian 
motion starting from c > 0. We define S = 1/Ty. 
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(a) Calculate the probability density function of S. What is the distribution 
of the random variable S$? 


(b) Find a number b (in terms of the constant c) for which we have P[S < b] 
= 0.5. 


Question no. 28 
The nonstationary Ornstein-Uhlenbeck process {X(t),t > 0} is a Gaus- 
sian process such that, if ø = 1 (see Subsection 4.2.5), 
dice ew zat 
EXA] = X(0)e"™ and V[X(t)] = = 


where a > 0 is a parameter. Suppose that X(0) = d > 0. We define Tp(d) = 
min{t > 0: X(t) = 0}. Calculate fr (a(t) when a = 1/2. 


Question no. 29 

Let {W(t),t > 0} be a Brownian motion with drift coefficient u and diffu- 
sion coefficient 7. We assume that the flow of a certain river can be modeled 
by the process {X(t),t > 0} defined by 


X(t) =eVOt® yt>0 


where k is a constant. Next, let d be a value of the flow above which the risk 
of flooding is high. Suppose that X(0) = d/3. Calculate the probability that 
the flow will reach the critical value d in the interval (0, 1] if p > 0 and o = 1. 


Question no. 30 | 
Let {X(t),¢ > 0} be an Ornstein—Uhlenbeck process for which a = 1 and 
o? = 2, and let {Y(t),t > 0} be the process defined by 


t 
Y(t) = i X(s) ds 
0 
Finally, we set 
Z(t) =X(t)+Y(t) fort >0 


(a) Calculate E/Z(t)]. 
(b) Calculate Cov[Z(t), Z(t + s)], for s,t > 0. 
Indication. We find that Cov|X (t), X(t + s)| = e7* and 


Cov[Y (t), Y(t + s)| = 2t — 1 +e™% — e78 +e7@*) for s,t > 0 


(c) Is the stochastic process {Z(t), t > 0} stationary? Justify. 
(d) Let Ta(z) := min{t > 0: Z(t) = d | Z(0) = z (< d)}. Calculate the 
probability density function of Ty(z). 
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Question no. 31 


Let {X(t),t > 0} be a Brownian motion with drift coefficient u and diffu- 


sion coefficient o?. 


(a) Suppose that = 0 and that a? is actually a random variable V having a 
uniform distribution on the interval (0,1). 

(i) Calculate E[X(t)] and Cov[X(s), X(t)], for s,¢ > 0. 

(ii) Is the stochastic process {X(t),t > 0} Gaussian and stationary? Jus- 
tify. 


(b) Suppose now that u > 0 and g? = 1. Let T-1,1 (£0) be the time the process 
takes to attain 1 or —1, starting from x € [—1, 1]. 

(i) It can be shown that m_1,1(zo) := E[T_1,1(xo)] satisfies the ordinary 
differential equation 


pa 


d 
D dag aleo) + p5—m-1,1(£0) = —1 


d£o 
Solve this differential equation, subject to the appropriate boundary condi- 
tions, to obtain m—1,ı (£0) explicitly. 

(ii) Similarly, the function p—1,1(£0) := P[X(T-1,ı(x0)) = 1] is a solution 
of À 

ld d 

———5 p — p11 (z0) = 0 
2 dae” 11 (20) + HEP 1,1(Zo) 


Obtain an explicit formula for p—1,ı (x0). 
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Poisson Processes 


5.1 The Poisson process 


We already mentioned the Poisson process in Chapters 2 and 3. It is a par- 
ticular continuous-time Markov chain. In Chapter 4, we asserted that the 
Wiener process and the Poisson process are the two most important stochas- 
tic processes for applications. The Poisson process is notably used in the basic 
queueing models. 

The Poisson process, which will be denoted by {N (t), t > 0}, is also a pure 
birth process (see Subsection 3.3.4). That is, N (t) designates the number of 
births (or of events, in general) that occurred from 0 up to time t. A process 
of this type is called a counting process. 


Definition 5.1.1. Let N(t) be the number of events that occurred in the inter- 
val [0, t]. The stochastic process {N(t),t > 0} is called a counting process. 


Counting processes have the following properties, which are deduced di- 
rectly from their definition. 


Properties. i) N(t) is a random variable whose possible values are 0,1,.... 


ii) The function N (t) is nondecreasing: N(t2) — N(t1) > 0 if te > ti > 0. 
Moreover, N (t2) — N (tı) is the number of events that occurred in the interval 
(ti, ta]. . 


Definition 5.1.2. A Poisson process with rate à (> 0) is a counting pro- 
cess {N(t),t > 0} having independent increments (see p. 50), for which 
N(0) =0 and 


N(r +t) ~— N(r) ~ Poi(At) V7,t > 0 (5.1) 
Remarks. We deduce from the preceding formula that a Poisson process also 


has stationary increments (see p. 50), because the distribution of N(7 + t) — 
N(r) does not depend on 7. Moreover, by taking 7 = 0, we may write that 
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N(t) = N(0 +t) — N(0) ~ Poi(At) Vt >0 (5.2) 


ii) If we specify in the definition that {N(t),t > 0} is a process with indepen- 
dent and. stationary increments, then we may replace the formula (5.1) by the 
following conditions: 


P[N(6) = 1] = 26 + 0(6) (5.3) 
P[N(6) = 0] = 1 — Aô + o(ô) (5.4) 


where o(d) is such that (see p. 125) 
l — 5.5 
lim > 0 ( ) 


Thus, the probability that there will be exactly one event in an interval of 
length 6 must be proportional to the length of the interval plus a term that is 
negligible if ô is sufficiently small. Furthermore, we have 


P(N(6) > 2] =1-{P[N(6) = 0] + P[N(6) =1]} =0(5) (5.6) 


Therefore, it is not impossible that there will be two or more events between 
an arbitrary tọ and tọ + 6. However, it is very unlikely when ô is small. 


It is not difficult to show that if N (ô) ~ Poi(Ad), then the conditions (5.3) 
and (5.4) are satisfied. We have 
(A8)? 


P[Poi(Ad) = 0] = e7% = 1 — AS + S tee 1 AS +08) (5-7) 


and 
P[Poi( AS) = 1] = Abe? = ASf1 — AS + 0(6)] = AS +.0(5) (5.8) 


As will be seen in Section 5.2, in the more general case where A = A(t), it can 
also be shown that if the conditions (5.3) and (5.4) are satisfied (and if the 
increments of {N(t),t > 0} are stationary), then the formula (5.1) is valid. 
Consequently, we have two ways of determining whether a given stochastic 
process is a Poisson process. 


Since the random variable V(t) has a Poisson distribution with parameter 
At, for all t > 0, we have 


E[N(t)] = At -= (5.9) 
E|N?(t)| = VIN(t)] + (EIND? = At +A? (5.10) 


As we did in the case of the Wiener process, we use the fact that the incre- 
ments of the Poisson process are independent (and stationary) to calculate its 
autocorrelation function. We may write, with the help of the formula (5.10), 
that 
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Ry (t,t + s):=E[N(t)N(t + s)] = EINN (t +s) — N(t)}] + E[N?) 
SEN (HEINE + s) — N(t)] + EIN?) 
= Atds + (At + A747) = A7t(t + 8) + AE (5.11) 


It follows that 
Cn(t,t+s) = Ry(t,t +s) — At[At + s)| = At (5.12) 


For arbitrary values of tı and te, the autocovariance function of the Poisson 
process with rate A > 0 is given by 


Cn (ti, te) = Amin{t,, t2} (5.13) 


Remarks. i) Note that the formula above is similar to that obtained for the 
Wiener process (see p. 178). Actually, the stochastic process {N*(t),t > 0} 
defined by 


N*(t) = N(t)—At fort >0 (5.14) 


has zero mean and an autocovariance function identical to that of a Wiener 
process with diffusion coefficient g? = à. We can also assert that the Poisson 
process, {N(t),t > 0}, and the Brownian motion with drift, {X(t),¢ > O}, 
have the same mean and the same autocovariance function if y = o? = À. 
Moreover, by the central limit theorem, we may write that 


Poi(At) ~ N(At, At) (5.15) 


if At is sufficiently large. 


ii) Since the mean of the Poisson process depends on the variable t, this process 
is not even wide-sense stationary (see p. 53), even though its increments are 
stationary. Furthermore, its autocovariance function depends on tı and on fa, 
and not only on |t2 — tj]. | 


Example 5.1.1. Suppose that the failures of a certain machine occur according 
to a Poisson process with rate À = 2 per week and that exactly two failures 
occurred in the interval [0,1]. Let tọ (> 3) be an arbitrary value of t. 

(a) What is the probability that, at time to, (at least) two weeks have elapsed 
since (i) the last failure occurred? (ii) the penultimate failure occurred? 

(b) What is the probability that there will be no failures during the two days 


beginning with to if exactly one failure occurred (in all) over the last two 
weeks? 


Solution. In order to solve a problem on the Poisson process, we must first 
determine the value of the parameter of the Poisson distribution for each 
question asked. Let N(t) be the number of failures in the interval [0,t], where 
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t is measured in weeks. Since the average arrival rate of the failures is equal 
to two per week, we have 
N(t) ~ Poi(2t) 


(a) (i) In this question, we are interested in the number of failures during a 
two-week interval. As the Poisson process has stationary increments, we can 
calculate the probability asked for by considering the random variable N(2) ~ 
Poi(4). We seek 


P|N(2) = 0] = e~* ~ 0.0183 (5.16) 


(ii) Two weeks or more have elapsed since the penultimate failure if and 
only if the number of failures in the interval {to — 2, to] is either 0 or 1. There- 
fore, we seek 

P[N(2) < 1] =e7*(1 +4) œ 0.0916 


(b) Since the Poisson process has independent increments, the fact that there 
has been exactly one failure over the last two weeks does not matter. We are 
interested in the value of 


N (to + 2/7) — N(to) = N(2/7) ~ Poi(4/7) 
We calculate 
P[N(2/7) = 0] = e74 ~ 0.5647 
Remarks. i) We could have written instead that A = 2/7 per day, and then, 
in (b), we would have had that N(2) ~ Poi(4/7). 


ii) In this problem, we assume that the rate of the Poisson process is the 
same for every day of the week. In practice, this rate is probably different on 
Sundays than on Mondays, for instance. It also probably varies at night from 
its value during the day. If we want to make the problem more realistic, we 
must use a parameter À that is not a constant, but rather a function of t, 
which will be done in Section 5.2. 


Proposition 5.1.1. Let {N,(t),t > 0} and {No(t),t > 0} be two independent 
Poisson processes, with rates 1 and àz, respectively. The process {N(t),t > 0} 
defined by 


N(t) = Ni(t)+ No(t) Vt>0 (5.17) 
is a Poisson process with rate À := A, + Ag. 
Proof. First, we have 
N(0) := N1 (0) + N2(0) =04+0=0 (5.18) 


as required. 
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Next, given that the increments of the processes {Ni(t),t > 0} and 
{No(t),é > 0} are independent, we can check that so are those of the pro- 
cess {N (t), t > 0}. We simply have to write that 


N (tk) — N (tk-1) = [Ni (tk) — Ni (tk-1)] + [No(te) — No(te-1)] (5.19) 


Finally, the sum of independent Poisson random variables, with parameters 
a@, and ag, also has a Poisson distribution, with parameter a@ := a, + ag. 
Indeed, if X; ~ Poi(a;), then its moment-generating function is given by (see 
Ex. 1.2.8) 


Mx, (t) = e~™ exp {ea;} (5.20) 
It follows that 
Mx,+x,(t) $ Mx, (t)Mx, (t) = e7 (+) exp {e(a +a2)} (5.21) 
Thus, we may write that 


N(T +t) — N(r) = [Ni(r + #) — Ny(r)] + [N2(r +t) — N2(7)] 
~ Poi((Ar +Ag}t) Yaro O (5.22) 


The preceding proposition can, of course, be generalized to the case when 
we add 7 independent Poisson processes, for 7 = 2,3,... . Conversely, we can 
decompose a Poisson process {N(t),t > 0} with rate A into j independent 
Poisson processes with rates A; (i = 1,2,...,7), where Ay +... +A; = A, 
as follows: suppose that each event that occurs is classified, independently 
from the other events, of type i with probability p;, for i = 1,... ,j, where 
pı +... + pj = 1. Let N,(t) be the number of type i events in the interval 
[0, t]. We have the following proposition. 


Proposition 5.1.2. The stochastic processes {Nj(t),t > 0} oe above are 
independent Poisson processes, with rates A; := Api, fori =1,... ,7. 


Proof. Since N(t) = oe N,(t), we may write that 
PING) = nissa NO) = t] 
= XO P(N (t) = m,- ,Nj(t) = ny | N(t) = k] PIN(t) = k] 


SPINU) = Ti tan »Nj(t) = n; |N@) =ni +... + n] 
x PIN(t) =ni +... +n; (5.23) 


Let n := nı +... + nj. The conditional probability above is given by 


P[Ni (6) = na,- N(= n ING) =n] = m G (5.24) 
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This is an application of the multinomial distribution (see p. 4), which gener- 
alizes the binomial distribution. It follows that 


À 
P{Ni(t) =ni... Nt) = 14) = Com alle p) -at (At)? r 


= I] on rpit Apit) (5.25) 
ni! 
i=1 

from which we find that 

P{N;(t) =n] = ma N fori =l; (5.26) 
Tli: 
and 
J 
P [N (t) = Nn], . , N;(t) = nj] = [[ PIM = nil (5.27) 
i=1 


Therefore, we can assert that the random variables N;(t) have Poisson distri- 
butions with parameters Ap;t, for i = 1,... ,7, and are independent. 

Now, given that {N (t), t > 0} has independent and stationary increments, 
the processes {N;(t),t > 0} have independent and stationary increments V i 
as well. It follows that N;(7 + t) — N;(T) ~ Poi(Ap,t), for i =1,... ,7. 

Finally, since N(0) = 0, we have that N; (0) = ... = N;(0) = 0. Hence, 
we may conclude that the processes {N;(t),t > 0} satisfy all the conditions 
in Definition 5.1.2 and are independent. O 


Remarks. i) We can use the other way of characterizing a Poisson process, 
namely the one when we calculate the probability of the number of events in 
an interval of length 6. We have 


P[N,(6) = 1] = P[Ni(6) =1| N(6) = 1] P[N() =] 
+P [N;(6) = 1] N(6) > 1] P[N(6) > 1] 
= pye*> 6 + 0(6) = p,[Ad + 0(6)] + 0(6) 
= Ap,d + o(ô) 


Moreover, we may write that 
P[N;(6) = 0] = 1 — Apjé + o(ô) (5.28) 
because N;(ô) < N(6), for all i, which implies that 
P [N;(6) > 2] < P[N(5) > 2] = o(ô) (5.29) 


We could complete the proof and show that the processes {N; ce t > 0} are 
indeed independent Poisson processes, with rates Ap;, fori =1,... ,7. 
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Ti 1, T3 Ty Ts Te t 


Fig. 5.1. Example of a trajectory of a Poisson process. 


ii) The fact than an arbitrary event is classified of type 1 must not depend on 
the time at which this event occurred. We will return to this case further on 
in this section. 


Since a Poisson process is a particular continuous-time Markov chain, we 
can assert that the time 7; that the process spends in state i € {0,1,...} 
has an exponential distribution with parameter v; > 0 (see p. 121) and that 
the random variables 79, T1, ... are independent (by the Markov property). 
Furthermore, because the Poisson process has independent and stationary in- 
crements, the process starts anew, from a probabilistic point of view, from 
any time instant. It follows that the 7;’s, for i = 0,1,..., are identically dis- 
tributed. All that remains to do is thus to find the common parameter of the 
random variables 7;. We have 


Pir > t] = P[N(t) = 0] = P[Poi(At) = 0] = e7% (5.30) 
= Jas < (l1—-e™“)=re™*_ fort > 0 (5.31) 


That is, 7 ~ Exp(à). Therefore, we can state the following proposition. 


Proposition 5.1.3. Let {N(t),t > 0} be a Poisson process with rate A, and 
let 7; be the time that the process spends in state i, fori = 0,1,.... The 
random variables To, T1, ... are independent and 7; has an exponential Exp() 
distribution, for all i. 


Notation. We designate by Ti, To, ... the arrival times of the events of the 
Poisson process {N (t), t > 0} (see Fig. 5.1). 


Corollary 5.1.1. In a Poisson process with rate À, the time needed to obtain 
a total of n events, from any time instant, has a gamma distribution with 
parameters œa = n and À. 
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Proof. The arrival time T,, of the nth event, from the initial time 0, can be 
represented as follows (because {N(t),t > 0} is a continuous-time stochastic 
process): 


n—ł 
Te yo me ASN (5.32) 
i=0 | 
Since the 7;’s are independent random variables, all of which having an ex- 
ponential distribution with parameter À, we can indeed assert that Tp ~ 


G(a = n, À) (see Prop. 3.3.6). Moreover, we deduce from the fact that the 
Poisson process has independent and stationary increments that 


Piles trig NG Jeo k HP, Toral k0 1.6 (5.33) 


That is, if we know that at a given time t, there have been exactly k events 
since the initial time, then the time needed for n additional events to occur, 
from tg, has the same distribution as Tn. O 


The preceding results and the following relation: 
NGen <= Test (5.34) 


provide us with yet another way of defining a Poisson process. ‘This alternative 
definition may be easier to check in some cases. 


Proposition 5.1.4. Let X; ~ Exp(A), fori = 1,2,..., be independent ran- 
dom variables, and let Ty := 0 and 


Ti Ne Gor ga O 33: (5.35) 
i=1 
We set 
N(t) = max{n > 0: Tn < t} (5.36) 


Then, {N(t),t > 0} is a Poisson process with rate X. 
Remarks. i) Since P{[T; > 0] = 1, we indeed have 
N(0) = max{n > 0: Tp <0} =0 (5.37) 
ii) We also have 
Nia. > AT St iin >t (5.38) 


from which we obtain the definition of N(¢) in terms of T, above. 


iii) To calculate the probability of the event {N(t) = n}, it suffices to notice 
that 
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P[N(t) =n] = PIN(t) > n] — PIN(t) > n +1] = Pi, < t|- Phy < t] 
(5.39) 


and to use the generalization of the formula (3.173) to the case when Y has 
a G(n, A) distribution: 


PIY < y] =1-— P|W <n-1])=P[W>n], where W ~ Poi(Ay) (5.40) 
We indeed obtain that 


PIN(t) = n] = PIT, < t] —P[Taya < t] (5.41) 
= P[Poi(At) > n] — P[Poi(At) > n + 1] = P[Poi(At) = n] 


iv) Let 
T% := min{t > 0: N(t) > d (> 0)} (5.42) 


If d € {1,2,...}, then we deduce from the relation (5.34) that Ty ~ G(d, A). 
When d ¢ {1,2,... }, we only have to replace d by [d] +1, that is, the smallest 
integer larger than d. Note that, since lim~o N(t) = oo, we may write that 
P[TF < co] = 1, for any real number d € (0, 00). 

A more difficult problem consists in finding the distribution of the random 
variable 


Teg := min{t > 0: N(t) > ct +d} (5.43) 
where c > 0 and d >Q. 


Example 5.1.2. Suppose that the random variables X; have a uniform distri- 
bution on the interval (0, 1], rather than an exponential distribution, in the 
preceding proposition. To obtain a Poisson process by proceeding as above, it 
suffices to define 


Y; = -5 ln X; Or BS ates 


Indeed, we then have, for y > 0: 
PLY; < y] = Plin X; > —dy] = P [X; > e™] = 1 — e7% 
so that 
fy.(y) =Ae*™” for y > 0 


Thus, the stochastic process {N (t),t > 0} defined by N(t) = 0, fort < Yı, 
and 


N(t) = max fn > 1: O¥ <e} fort > Y (5.44) 


t=] 


is a Poisson process with rate À. 
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Remark. If we define Yo = 0, then we may write that 


NG) = max {n> 0: JY, <t) Vti>0 (5.45) 
i=0 


Proposition 5.1.5. Let {N(t),t > 0} be a Poisson process with rate A. We 
have that Tı | {N(t) = 1} ~ U(0,t], where Tı is the arrival time of the first 
event of the process. 


Proof. For 0 < s < t, we have 


Pity<s|N()=4 = PASS NO= 


PIN() =1 
_ PIN(s) = 1, N(t) — N(s) = 0] 
-= PING) = 1 
ind. (Ase~># eA) 8s 
e =O 


Remarks. i) The result actually follows from the fact that the increments of 
the Poisson process are independent and stationary. This indeed implies that 
the probability that an event occurs in an arbitrary interval must depend only 
on the length of this interval. 


ii) More generally, if T* denotes the arrival time of the only event in the 
interval (t,,t2], where 0 < tı < te, then T* is uniformly distributed on this 
interval. 


iii) The random variable T, | {N(t) = 1} is different from Ty := Ty | {Ti < t}. 
The variable Tř has a truncated exponential distribution: 
PIT; < s] 1 — eàs 


Ply Se= PR ss| T sts paza = Toe forO<s<t 
i < n 
(5.46) 


Note that the occurrence of the event {N(t) = 1} implies the occurrence of 
{T; < t}. However, {T, < t} => {N(t) > 1}. On the other hand, we may 
write that 


T, |{N() =1} =T% | {Th <t,T2 >t} (5.47) 


We would like to generalize the preceding proposition by calculating the 
distribution of (7),... ,7;,), given that exactly n events occurred in the in- 
terval (0,t]. Let us first consider the case when n = 2. Let 0 < ty < t2 <t. 
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To obtain the conditional distribution function of the random vector (T1, T2), 
given that N (t) = 2, we begin by calculating 
PIT, < ti, To < t2, N(t) = 2] 
= PIN (ti) = 1, N(t2)— N (t1) = 1, N(t) — N(t2) = 0] 
+PIN (t1) = 2, N(t) — N(t1) = 0] 


2 
= (Ate Ate za je Sorte NG- 2) af (ce 
2 
— (Malt = ty) ae oT) ert (5.48) 


where we used the fact that the increments of the process {N(t),t > 0} are 
independent and stationary, from which we find that 


PIT; < ti, To < te, N(t) = 2 
PIN(t) = 2] 

(N7t1 (to — t1) + 4(At1)?) e7% 

Qty (tg —t1) + ti 

Sann athe 


PIT; < t1,T2 < tg | N(t) =2) = 


(5.49) 
so that 
0? [2al — tı) i a) 


ti, ta | 2) = —— 72 
FT, TiNa (t1, t2 | 2) DE, Os 2 72 


2 2 
gt a oD forO0 <ty <to<t (5.50) 


Remark. From the preceding formula, we calculate 
"3 2(t — ty) 
Frnt) lti | 2) = 72 dt = a for 0 < ti <t (5.51) 
i 


Note that the distribution of Tı | {N (t) = 2} is not uniform on the interval 
(0, t], contrary to that of T, | {N (t) = 1}. We also have 


A 2t 
frina (t2 | 2) = OP dii = rH fr <t <t (5.52) 
Finally, we may write that 
Jr nina (t t2 | 2) Di? 
frit, (te | ti, 2) = + = a 





I 
aE for0 <t; <tg<t (5.53) 
— ti 
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That is, T2 | {71 = t, N(t) = 2} ~ U(tı,t], from which we deduce that 
(Tz — Ti) | {71 = t1, N(t) = 2} ~ U(0,t — ty]. 

To obtain the formula in the general case, we can consider all the possible 
values of the random variables N(t;), N(t2) — N(t1), .-., N(tn) — N(tn-1), 
given that T1 < t1, T2 < t2,... , Tn < tn, N(t) = n. The total number of ways 
to place the n events in the intervals (0, t1], (t1,t2], .-., (tn—1,tn] is given 
by the multinomial coefficients (see p. 4). Moreover, N(t1) must be greater 
than or equal to 1, N (t2) must be greater than or equal to 2, etc. However, 
it suffices to notice that the only nonzero term, after having differentiated 
the conditional distribution function with respect to each of the variables 
ti,... tn, will be the one for which the event 


F={N(t1) =1}N{N(te) — N(t1) =1}9...N{N(tn) — N(tn-1) = 1} 
(5.54) 


occurs. Let 

G= {N(t1) > 1} N{N(te) = 2}N...N{N(tr) = n} (5.55) 
We have 
P |T; < t1, T2 < te,... , Tn Stn, N(t) = n] 

= P[F N {N(t) =n}] + P[GN{N(t) =n} 

= P|FN{N(t)—- N(tn) = 0}] + PIGN{N(Et) = n} 


n 
— Atje ™ x I] A(tk = tpai OR? x e7 A(t-tn) 
k=2 
+P(GN{N(t) =n} 
Ên 
= A"tye™™ [| (te — tk-1) + PIGN {N(t) = n} (5.56) 
k=2 


It follows that 


PIT; < tj, lo < to, s«. In <ty | N(t) =n] 
_ PIT <tr Ta S lnc In < tn, N(t) =n] 
Oo PNAS 
_ Atie ™ [Tena (te = tr-1) 


(At) re ^t jn! +P[G|N(t)=n] (5.57) 


Since at least one t; is not present in the term P[G | N(t) = n}, we may write 
that 


ÍT, Tec Tal N (t) (ti, to, Boe stn n) 
Oo” tı [pao (te — te-1) 


~ Ot Oty --- Ot, tn jn! 
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n! 
E for 1p = to aiaa (5.58) 


We have proved the following proposition. 


Proposition 5.1.6. Let {N(t),t > 0} be a Poisson process with rate À. Given 
that N(t) = n, the n arrival times of the events, T;,..., Ta, have the joint 
probability density function given by the formula (5.58). 


Remarks. i) Another (intuitive) way of obtaining the joint density function of 
the random vector (7;,... , Tn), given that N(t) = n, is to use the fact that, 
for an arbitrary event that occurred in the interval (0,t|, the arrival time of 
this event has a uniform distribution on this interval. We can consider the n 
arrival times of the events as being the values taken by n independent U(0, t] 
random variables U; placed in increasing order (since Ty < To < ... < Th). 
There are n! ways of putting the variables Ui, ..., Un in increasing order. 
Moreover, by independence, we may write, for 0 < ti < tg <... < tn St, 
that 


n n 
1 1 
fuy,... Un (ty +++ stn) = TEZO = JI ree (5.59) 
71 i=1 
Thus, we retrieve the formula (5.58): 
n! 
fr TAINO stn (0) = Morale ste) =E (5.60) 
f0<t eta anain E 
ii) When we place some random variables X1, ..., Xn in increasing order, 
we generally use the notation X(1),...,X(n), where X < Xg ifi < J. In 
the continuous case, we always have that Xq) < Xg) ifi < j. The variables 
placed in increasing order are called the order statistics of X1,...,Xn. We 


have that Xa) = min{X1,...,X,} and Xin) = max{Xj,... , Xn} 


Let us now return to the problem of decomposing the events of a Poisson 
process, {N (t), t > 0}, into two or more types. 


Proposition 5.1.7. Suppose that an event of a Poisson process with rate À, 
{N (t), t > 0}, that occurs at time s is classified, independently from the other 
events, of type i with probability p;(s), where i = 1,... ,j and X2; pi(s) = 1. 
Let N;(t) be the number of type i events in the interval |0, t]. The N;(t)’s are 
independent random variables having Poisson distributions with parameters 


t 
Ajea à | pis) ds forts L2 sia (5.61) 
0 
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Proof. We know that the arrival time of an arbitrary event that occurred in 
the interval (0, t] has a uniform distribution on this interval. Consequently, we 
may write that the probability p; that this event is of type i is given by 


t 
p= | pi(s)= ds for i=1,2,... 3 (5.62) 
0 


Proceeding as in the proof of Proposition 5.1.2, we find that the random 
variable N;(t) has a Poisson distribution with parameter 


t 
A(t) =Apit= af p(s) ds fori=1;2, i7 (5.63) 
0 


and that the variables N;(t), N2(t), ..., N;(t) are independent. 0 


Remark. The random variables N;,(t) have Poisson distributions. However, 
the stochastic processes {Nj(t),t > 0} are not Poisson processes, unless the 
functions p;(s) do not depend on s, for all i. 


Example 5.1.8. Customers arriving at a car dealer (open from 9 a.m. to 9 p.m.) 
can be classified in two categories: those who intend to buy a car (type I) and 
those who are just looking at the cars or want to ask some information (type 
II). Suppose that 


1/2 from 9 a.m. to 6 p.m. 


P|Customer is of type I] = eo from 6 p.m. to 9 p.m. 


independently from one customer to another and that the arrivals constitute 
a Poisson process with rate À per day. 

(a) Calculate the variance of the number of type I customers arriving in one 
day if A = 50. 

(b) Suppose that the average profit per car sold is equal to $1000 and that 


A = 10. What is the average profit for the dealer from 9 a.m. to 6 p.m. ona 
given day, knowing that at least two cars were sold during this time period? 


Solution. (a) Let N7(t) be the number of type I customers in the interval (0, t}, 
where ¢ is in (opening) hours. We can write that 


N1(12) ~ Poi (ZJ pr(s) is) = Poi(21.875) 
9 


21 18 1 2i 1 
f pr(s) ds = f = ds+ | ~ ds = 21/4 
9 9 2 ig 4 


We seek V[N;(12)] = 21.875. 


(b) Let Ny(t) be the number of sales in t days, from 9 a.m. to 6 p.m. The 
process {Nj (t), t > 0} is a Poisson process with rate 


because 
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9 T 
isa 22 
* 12% 93 


We are looking for 1000 E[N: (1) | Ni (1) > 2] := 1000x. We have 


= 3.75 


E(Ni(1)] = 0 + E[N: (1) | M (1) = UPIM (1) = 1] + eP[Ni(1) 2 2] 
It follows that 


3.75 = 1 -3.7503 T5 4z [1 —3-7(1 43,75) 


We find that x ~ 4.12231, so that the average profit is given by (approxi- 
mately) $4122.31. 


Proposition 5.1.8. Let {N,(t),t > 0} and {No(t),t > 0} be independent 
Poisson processes, with rates 41 and àa, respectively. We define the event 
Frino = nı events of the process {N(t),t > 0} occur before nz events of the 
process {N2(t),t > 0} have occurred. We have 


P|Fain.| = PIX > ni], where X ~ B(n = nı +n — l, p := sats) 








Ait 
(5.64) 
That 1s, 
PIF i 7 — ni +n — Ll i i Jo nı +tnz—l—i 
GE i A tà) A tA 
(5.65) 


Proof. Let N (t) := Nı (t) + N2(t), and let Æ; be the random experiment that 
consists in observing whether the jth event of the process {N (t), t > 0} is an 
event of the process {Ni (t), t > 0} or not. Given that the Poisson process has 
independent and stationary increments, the E;’s are Bernoulli trials, that is, 
independent trials for which the probability that the 7th trial is a success is 
the same for all 7. Then the random variable X that counts the number of 
successes in n trials has, by definition, a binomial distribution with parameters 
n and (see Prop. 3.3.4) 


Al 


p := PTa < T21| = y x NG 


(5.66) 





where Ti ~ Exp(A1) and Tz ı ~ Exp(A2) are independent random variables. 
Since Fy, na occurs if and only if there are at least nı events of the process 
{N (t), t > 0} among the first ny + nz — 1 events of the process {N(t),t > 0}, 
we obtain the formula (5.65). A 
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Example 5.1.4. Let {X(t), 
t 


(t),0 <t < 1} be the stochastic process defined from 
a Poisson process {N (t), 


0 
> 0} with rate as follows: 
X({t)= N(t)-—tN(1) forO<t<1 


Note that, like the Brownian bridge (see p. 196), this process is such that 
X (0) = X(1) = 0. We calculate 


E|X(t)| = E[N(t) —tN(1)] = E[N] — tE[N(1)] = At — t(A- 1) = 0 
for all t € [0,1]. It follows that if 0 < t; < 1, for i = 1,2, then 


Cx (ti, t2) = E[X(t1)X(t2)] — 0? 
= E {[N (t1) — t1N(1)] [N(t2) — teaN(1)}} 
= E|[N(ti)N(te)| — t E [N(1)N (t2)] — 2B [N (t1) N (1) 
+ tyt2E[N?(1)] 
= tito + A min{ty, t2} — ty (Ata + Aty) — t2 (A7t1 + Afr) 
+ tita (A? +A) 
= Amin{t;,to} — Atite 


where we used the formula 
Ryn (th, t2) = Cn (th, t2) + E[N (tti ELN (t2) =À min{t,, t2} + X tite 


We can generalize the process above by defining {X(t),0 < t < c} by 
X(t) = N(t) - EN No hinete 


where c is a positive constant. 


Example 5.1.5. We know that the Poisson process is not wide-sense stationary 
(see p. 232). On the other hand, its increments are stationary. Consider the 
process {X (t), t > 0} defined by 


X(t)= N(t+c)—N(t) fort >0 
where c is a positive constant and {N(t),t > 0} is a Poisson process with rate 
A. Note that X(0) = N(c) ~ Poi(Ac). Thus, the initial value of the process is 
random. We have 


N(t+c)—N(t)~ Poi(Ac) => E[X(t)|=Ac for allt >0 


Next, by using the formula for Ry(t,,t2) in the preceding example, we 
calculate (for s,t > 0) 


Rx (t,t +s) := E[{N(t+c) -N@®HN(t+s+c)—-N(t4+s)}] 
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= E|IN(t+e)N(t+s+c)]-E[N(t+c)N(t+s)] 
— E[N(HYN (t + s +c) + EINGN(t+4+ 8) 
= A[(t+c)—t-— min{c,s}—t+t] 
+A [EHe +s +e) (ttet s)-tlt+s +e) +t s) 
= A(e— min{e, s}) + Ae? 


It follows that 


Cx (t,t+s):= Rx(t,t+s) — ELX(t)|E[X(t+ s)| 
= \(e~ min{e, s}) +X P — (Ac)? 


0 ifc<s 
M Ora) m ifc>s 


We can therefore write that Cx(t,t + s) = Cx(s). Thus, the stochastic 
process {X(t),t > 0} is wide-sense stationary (because E[X (t)] = Ac), which 
actually follows from the stationary increments of the Poisson process. More- 
over, the fact that the autocovariance function C'x(s) is equal to zero, for 
c < s, is a consequence of the independent increments of {N (t), t > 0}. 


Example 5.1.6. We suppose that the number N (t) of visitors of a certain Web 
site in the interval [0,t] is such that {N(t),t > 0} is a Poisson process with 
rate A per hour. Calculate the probability that there have been more visitors 
from 8 a.m. to 9 a.m. than from 9 a.m. to 10 a.m., given that there have been 
10 visitors from 8 a.m. to 10 a.m. 


Solution. Let N; (respectively, Nə) be the number of visitors from 8 a.m. 
to 9 a.m. (resp., from 9 a.m. to 10 a.m.). We can assert that the random 
variables N; ~ Poi(5) and N2 ~ Poi(5) are independent. Furthermore, we 
have N; | {Ni + No = 10} ~ B(n = 10,p = 1/2), for i = 1,2. We seek the 
probability x := P|N; > Nə | Ni+N2 = 10). By symmetry, we can write that 


10 
=x + P(N, = No| Ni + Nz = 10] +£ = 2z + (is Jaa” ~ 2g + 0.2461 
Then x œ 0.3770. 


Example 5.1.7. We define M(t) = N(t) — (t?/2), for t 
t > 0} is a Poisson process with rate A. Calculate P/{1 
Sı := min{t > 0: M(t) > 1}. 


Solution. The increments of the Poisson process being independent and sta- 
tionary, we can write that 


Pil < S$, < V2] 
= P[N(1) = 0, N(V2) — N(1) > 2] + PIN(1) = 1, N(vV2)- N(1) > 1] 
igen 1 — e7 (V2-1)A (1 2a = 1),)| + re E = 2a 


0, where {N (t), 


= 0, 
< Sı < v2], where 
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Fig. 5.2. Example of a trajectory of a telegraph signal. 


5.1.1 The telegraph signal 


As we did with the Brownian motion and in Examples 5.1.4 and 5.1.5, we can 
define stochastic processes from a Poisson process. An interesting particular 
transformation of the Poisson process is the telegraph signal {X(t),t > 0}, 
defined as follows: 


nN 1 if N(t) = 0,2,4,... 
AES a if N(t) =1,3,5,... (5.67) 


An example of a trajectory of a telegraph signal is shown in Fig. 5.2. 


Remark. Note that X(0) = 1, because N(0) = 0. Thus, the initial value of the 
process is deterministic. To make the starting point of the process random, 
we can simply multiply X(t) by a random variable Z that is independent of 
X(t), for all t, and that takes on the value 1 or —1 with probability 1/2. It is 
as if we tossed a fair coin at time t > 0 to determine whether X(t) = 1 or —1. 

The process {Y (t),t > 0}, where Y(t) := Z - X(t), for all t > 0, is called 
a random telegraph signal. We may write that Z = Y(0). Moreover, to be 
precise, we then use the expression semirandom telegraph signal to designate 
the process {X(t),t > 0}. We already encountered the random telegraph 
signal in Example 2.3.2. 


To obtain the distribution of the random variable X(t), it suffices to cal- 
culate 





oo OO 2k 
P[X(t) =1] = )— P[N(t) = 2k] = Sree A 
k=0 k=0 ee) 
AE goat —2Mt 
= Me = teL Vt>0 (5.68) 
because 
ert + e7 àt oO (At)?* 
coshAt := B ` (Sk) (5.69) 
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where “cosh” denotes the hyperbolic cosine. It follows that 


1 + e7 2At ji a e7 2At 
i ae 
In the case of the process {Y (t), t > 0}, we have 


P[X(t) = —lJ =1- (5.70) 


PlY(t)=1] = PIZ- X(t) =1] 
= PIX(t)=1|Z =1]P[Z = 1] 
+ PIX(® = -1 | Z = -1]PÍZ = —1] 


S. - {P[X(t) = 1] + P[X(t) = -1]} = > (5.71) 


z 


Thus, P[Y (t) = 1] = P[Y (t) = —1] = 1/2, for all t > 0, so that E[Y (t)| = 0, 
whereas 


E|X(t)] =1-——S— He HH 


yt>0 (5.72) 


- To obtain the autocorrelation function of the (semirandom) telegraph sig- 
nal, we make use of the definition X(t) = (—1)*™. If s > 0, we may write 
that 


Rx(t,t+s) := E[X()X(t+s)} = E k-19” Ops a 
ap (-1y240(-1) Mer | mE: g ra] 
=E kgr] = E[X(s)] =e"? (5.73) 
where we used the fact that the increments of the Poisson process are station- 
ary. We then have 


Cx(t,t + 8) = e799 — Pte Ales) = e248 (1 eM) Wt > 0 
(5.74) 


Finally, since 


Y(t)Y(t+s):= [Z-X(t)[Z-X(t+s)] =Z?-X(t)X(t + $) 
= 1. X(t) X(t +s) =X(t)X(t4+s) (5.75) 


we find that 


Cy (t,t +s) = Ry(t,t+s8s)= Ry(t,t+s)=e°™ Vs,t>0 (5.76) 

We deduce from what precedes that the process {Y (t), t > 0} is wide-sense 
stationary. As we mentioned in Example 2.3.2, it can even be shown that it is 
strict-sense stationary. On the other hand, {X (t), t > 0} is not WSS, because 
its mean depends on t (and Cx (t,t +s) # Cx(s)). Furthermore, in Example 
2.3.2, we showed that the random telegraph signal is a mean ergodic process. 
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5.2 Nonhomogeneous Poisson processes 


In many applications, it is not realistic to assume that the average arrival 
rate of events of a counting process, {N(t),¢ > 0}, is constant. In practice, 
this rate generally depends on the variable t. For example, the average arrival 
rate of customers into a store is not the same during the entire day. Similarly, 
the average arrival rate of cars on a highway fluctuates between its maximum 
during rush hours and its minimum during slack hours. We will generalize the 
definition of a Poisson process to take this fact into account. 


Definition 5.2.1. Let {N(t),t > 0} be a counting process with independent 
increments. This process is called a nonhomogeneous (or nonstationary ) 
Poisson process with intensity function X(t) > 0, for t > 0, if N(0) =0 
and 

i) PIN(t + 6) — N(t) = 1] = A(t) + o(ô), 

ti) PIN(t + 6) — N(t) > 2] = o(6). 


Remark. The condition i) implies that the process {N(t),t > 0} does not have 
stationary increments unless A(t) = > 0. In this case, {N (t), t > 0} becomes 
a homogeneous Poisson process, with rate À. 


As in the particular case when the average arrival rate of events is constant, 
we find that the number of events that occur in a given interval has a Poisson 
distribution. 


Proposition 5.2.1. Let {N(t),t > 0} be a nonhomogeneous Poisson process 
with intensity function A(t). We have 


NGL NOW nimen Vetse Gan 


where 
m(s) := [ A(T) dr (5.78) 


Proof. Let 
Pn(s,t) := P[N(s+t)—N(s)=n] forn=0,1,2,... (5.79) 


Using the fact that the increments of the process {N(t),t > 0} are indepen- 
dent, and using the two conditions in the definition above, we may write, for 
n = 1,2,..., that 


Pn(s,t + 6) 
= P|N(s +t) — N(s) =n, N(s+t+6)—N(s+t) =9] 
+ PIN(s+t)—N(s)=n—1,N(s+t+6)—N(s+t) =1]+0(6) 
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= pn(s,t)[1 — A(s + t)6 + 0(6)] + pn_-i(s,t)[A(s +t) + o(ô)] + o(8) 
from which we find that 
Pn{s,t +6) — pn(s,t) = Als + t)d[pp—1(s, t) — pn(s,t)| + o(6) (5.80) 


Dividing both sides by 6 and taking the limit as 6 decreases to zero, we obtain 


Z onl’ t) 7 A(s T t)[Pn-1 (s, t) B Pn (S, t)| (5.81) 


When n = 0, the equation above becomes 


O 
A, Pols: t) = —A(s + t)po(s, t) (5.82) 


The variable s may be considered as a constant. Therefore, this equation is 
a first-order, linear, homogeneous ordinary differential equation. Its general 
solution is given by 


po(s,t) = co exp {- m A(T) ir} (5.83) 


where co is a constant. Making use of the boundary condition po(s,0) = 1, we 
obtain cp = 1, so that 


s+t 
po(s,t) = exp f- A(T) ir} = emls)—mstt) for s,t > 0 (5.84) 


Substituting this solution into Eq. (5.81), we find that 


ð 
5 Puls, t) = A(s + t) Rra — pr(s,t)| (5.85) 


We may rewrite this equation as follows: 


< p(s, t) = ene ae) — pı (s, t) Z [m(s + t) — m(s)] (5.86) 


We easily check that the solution of this nonhomogeneous differential equation, 
which satisfies the boundary condition p;(s,0) = 0, is 


p1(s,t) = e™)-™(s+4) ims +4) —m(s)| for s,t >0 (5.87) 


Finally, we can show, by mathematical induction, that 


Dn(s, t) = ern(s)-m(ose) S +t) — mle)’ y St > 0 and m = 0, L roe D 
n: 
(5.88) 
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Remarks. i) The function m(t) is called the mean-value function of the process. 


ii) Suppose that A(t) is a twice-differentiable function such that A(t) < A, 
for all t > 0. We can then obtain a nonhomogeneous Poisson process with 
intensity function A(t) from a homogeneous Poisson process {N;(t),t > 0}, 
with rate A, by supposing that an event that occurs at time t is counted with 
probability A(t)/A. Let | 

F = exactly one of the events that occur in the interval (¢,t+.4] is counted, 
and let 

N(t)= the number of events counted in the interval [0, t]. 
We have (see Prop. 5.1.7) 


PIN(t +6) —N(t)=1)= 3 PI{N,(t +5) — Ni (t) =k} NF] 
k=1 


=o 1 gi A Hie) 


t 
MEF ED to 


for some c € (0,1), by the mean-value theorem for integrals. Since 


= [Ad + 0(4)| (8) (5.89) 


A(t + cd) = A(t) + cdX'(t) + o(ô) (5.90) 
we may write that 
P(N(t + 6) — N(t) = 1] = [A6 + 0(6)] e +o(ô) 
= A(t)d + o(ô) (5.91) 
iii) If we assume that 
i A(t) dt =œ for all tọ € [0, 00) (5.92) 


to 
then the probability that at least one event will occur after tg is equal to 1: 


dim PLN (to +s)—N(to) > 1) =1- lim PIN (to +s) — N(to) = 0] 


to+s 
= 1 — lim exp {~ f A(t) it} 


to 


=1-—-0=]1 (5.93) 
Note that the formula (5.92) is valid when A(t) = A > 0. 


Let Tı be the random variable that denotes the arrival time of the first 
event of the process {N(t),¢ > 0}. We will now calculate the distribution of 
Tı, given that N(t) = 1, as we did in the case of the homogeneous Poisson 
process (see Prop. 5.1.5). 
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Proposition 5.2.2. Let {N(t),t > 0} be a nonhomogeneous Poisson process 
with intensity function A(t). The probability density function of the random 
variable S := Tı | {N(t) = 1} is given by 

A(s) 

fis(s) mab forO<ss (5.94) 
Proof. If 0 < s < t, we may write that 


Prenat a a O 





PING) =| 
a (m(s)e~™(s)) e~m(t)+m(s) m(s) 
= m(t)e—™t) ~ m(t) 
Since 
d d f° 
z ms) = zj A(u) du = A(s) (5.95) 


we obtain the formula (5.94). 0 


Remark. The formula for the distribution function of the arrival time S of the 
single event that occurred in an arbitrary interval (7,7 + t] is 


m(s) — m(7) 


ae) = m(T +t) —m(r) 


forO<Tr<s<Tti (5.96) 


so that 


A(s) 
= ——__———. for0<Tr<s< t 0.97 
fs(s) = ae E or 0<7r<s<sT+ (5.97) 
where S := T | {N(r +t) — N(r) = 1} and T is the arrival time of the first 
event of the nonhomogeneous Poisson process in the interval (7,7 + tl. 


Example 5.2.1. Let {N(t),¢ > 0} be a nonhomogeneous Poisson process with 
intensity function A(t) > 0, for t > 0. We set 


M(H) =N (mtt) for allt > 0 


where m7! (t) is the inverse function of the mean-value function of the process. 
Remark. This inverse function exists, because we assumed (in this example) 
that the intensity function A(t) is strictly positive, for all t, so that m(t) is a 
strictly increasing function. 
We then have 
M(0) = N (m7*(0)) = N(0) =0 
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Moreover, we calculate, for all 7,t > 0 and n =0,1,..., 
P[M(r +t) — M(r) =n] = P [N (mt (r +t)) -N(m7"(7)) =n] 
= P [Poi (m (m~t (r + t)) —m(m7*(r))) =n] 
= P[Poi(r +t — T) = n] = P[Poi(t) = n] 


Finally, since the function m~! (t) is strictly increasing, we may assert that 
the stochastic process {M(t),t > 0}, like {N (t),t > 0}, has independent in- 
crements. We then deduce, from what precedes, that {M (t), t > 0} is actually 
a homogeneous Poisson process, with rate A = 1. 


5.3 Compound Poisson processes 


Definition 5.3.1. Let X1, X2,... be independent and identically distributed 
random variables, and let N be a random variable whose possible values are 
all positive integers and that is independent of the X;,’s. The variable 


N 
Sv := > Xp (5.98) 


is called a compound random variable. 


We already gave the formulas for the mean and the variance of Sy [see 
Eqs. (1.89) and (1.90)]. We will now prove these formulas. 


Proposition 5.3.1. The mean and the variance of the random variable Sy 
defined above are given, respectively, by 








| yx = E(NJE[X,] (5.99) 
and 
N 
V b3 x = E[N]V [X1] + VIN](E[X1])? (5.100) 
k=İ 
Proof. First, we have 
E ed = 5 BX = nE[X:] (5.101) 
= k=1 





Then, since N is independent of the X;’s, we may write that 


N 
E |X X| N= 
k=1 


n 


Ps |- = nE[X] (5.102) 
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so that 





N 
E X Xk 
k=1 





N = NE[X,] (5.103) 
It follows that 

N 

k=1 


Remark. It is not necessary that the random variables X, be independent 
among themselves for the formula (5.99) to be valid. 








xi] - E[NE(X\]] = BINIE[X) (5.104) 


Next, proceeding as above, we find that 


N 
Vy GINS 
k=1 


With the help of the formula (see p. 28) 


= NV[X,] (5.105) 








N 
S| XklN 


k=] 








“| Hid nVIXy] = V 





V[Sn] = E[V[Sy | N]] + VIEIS |N] (5.106) 


we may then write that 





v [oa = E[NV[X,]] + VINE[X]]] 


= E(N|V[X1} + (E[X4))°V[N] 0 (5.107) 


Definition 5.3.2. Let {N(t),t > 0} be a Poisson process with rate A, and let 
X1,X2,... be random variables that are i.i.d. and independent of the process 
{N(t),t > 0}. The stochastic process {Y(t),t > 0} defined by 


Nit) 
= DR Xk VWt>0 (andY(t)=0 if N(t) = 0) (5.108) 
k=1 


ts called a compound Poisson process. 


Remarks. i) This is another way of generalizing the Poisson process, since 
if the random variables X; are actually the constant 1, then the processes 
{Y (t),t > 0} and {N(t),t > 0} are identical. 

ii) A Poisson process, {N(t),t > 0}, only counts the number of events that 
occurred in the interval [0, t], while the process {Y (t), t > 0} gives, for exam- 
ple, the sum of the lengths of telephone calls that happened in (0, t], or the 
total number of persons who were involved in car accidents in this interval, 
etc. Note that we must assume that the lengths of the calls or the numbers 
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of persons involved in distinct accidents are independent and identically dis- 
tributed random variables. We could consider the two-dimensional process 
{(N(t), Y(t)),t > 0} to retain all the information of interest. 


Using Proposition 5.3.1 with Sy = Y(t) and N = N(t), we obtain 
ElY (t)| = EIN (G) EIX] = AEX] | (5.109) 
and 


VIY (E) = EINO [Xa] + VIN()|(E))? 
= At (V[X1] + (E[X4])*) = AE[X7] (5.110) 


We can easily calculate the moment-generating function of the random 
variable Y (t). Let 


M,(s) = Mx,(s) := E [e®**] (5.111) 
We have 


My (1)(s):=E Gae Ele ee) =E |E ettn ee) | 


ind. De E [esit +Xn IJe a i.i.d. Sm (se~ A" 
n=O 


n! 
= eM1(9)At — exp{At[M: (s) — 1)} (5.112) 


This formula enables us to check the results obtained above. In particular, 
we have 


EY ()] = © Myeo(s)|g = Mro (S)AMI C), 


= Myqy(0)AtM{(0) = 1MtE[Xy] = MELXy) (5.113) 


When _X, is a discrete random variable, whose possible values are 1,2,... ,7, 
we may write that 


Y(t) = 5 iN;(t) (5.114) 


i=1 


where N;(t) is the number of random variables X}, (associated with some 
random events) that took on the value 7 in the interval [0, t]. By Proposition 
5.1.2, the processes {N; a t > 0} are independent Poisson processes with 
rates Apx,(t), for i =1,... ,7. 


Remarks. i) This representation of the process {Y (t), t > 0} can be generalized 
to the case when X; is an arbitrary discrete random variable. 


5.3 Compound Poisson processes 257 


ii) The moment-generating function of the random variable Y(t) in the case 
above becomes 


My (s) = exp Dx Die 1) px,( @} (5.115) 


Since limzoo N(t) = œ, we deduce from the central limit theorem the 
following proposition. 


Proposition 5.3.2. For t sufficiently large, we may write that 


Y(t) ~ N (ME[X:], AE[X7]) (5.116) 


Remark. For the approximation to be good, the number of variables in the 
sum must be approximately equal to 30 or more (or maybe less), depending 
on the degree of asymmetry of (the distribution of) the random variable X: 
with respect to its mean. 


Finally, let {Yi(t),t > 0} and {Y2(t),t > 0} be independent compound 
Poisson processes, defined by 


N; (t) 
th= S° Xin Vt>O0 (and Y;(t) =0 if N,(t) = 0) (5.117) 
k=1 


where {N;(t),t > 0} is a Poisson process with rate X;, for i = 1,2. We know 
that the process {N(t),t > 0}, where N(t) := Nit) + No(t) Vt > 0, isa 
Poisson process with rate À := A; +A 2 (because the two Poisson processes are 
independent; see Prop. 5.1.1). Let X; be the random variable associated with 
the kth event of the process {N (t), t > 0}. We may write that 





Xı k with probability p := 
Xn ATOR (5.118) 


Xə k with probability 1 — p 


(That is, X, has the same distribution as X1 k (respectively, Xo,,) with prob- 
ability p (resp., 1 — p).) Thus, we have 


P[X, <a] = P|Xik < zip + P|X2k < z](1—p) (5.119) 


Since the random variables X1, Xo,... are i.i.d., and are independent of the 
Poisson process {N (t),t > 0}, we may assert that the process {Y (t),t > 0} 
defined by 


Y(t) =Y (t) + Yo(t) fort >0 (5.120) 


is a compound Poisson process as well. 
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Example 5.3.1. Suppose that the customers of a certain insurance company 
pay premiums to the company at the constant rate œ (per time unit) and 
that the company pays indemnities to its customers according to a Poisson 
process with rate À. If the amount paid per claim is a random variable, and if 
the variables corresponding to distinct claims are independent and identically 
distributed, then the capital C(t) at the company’s disposal at time t is given 
by 
C(t) = C(0) + at — Y(t) 


where {Y (t), t > 0} is a compound Poisson process (if, in addition, the indem- 
nity amounts do not depend on the number of indemnities paid). An important 
question is to be able to determine the risk that the random variable C(t) will 
take on a value smaller than or equal to zero, that is, the risk that the com- 
pany will go bankrupt. If a < Aw, where u := E[X1] is the average indemnity 
paid by the company, then we find that the probability that it will eventually 
go bankrupt is equal to 1, which is logical. 


5.4 Doubly stochastic Poisson processes 


In Section 5.2, we generalized the definition of a Poisson process by allowing 
the average arrival rate of events of the process to be a deterministic function 
A(t). We will now generalize further the basic Poisson process by supposing 
that the function A(t) is a random variable A(t). Thus, the set {A(t),t > 0} 
is a stochastic process. For this reason, the process {N(t),t > 0} is called a 
doubly stochastic Poisson process. First, we consider the case when the random 
variable A(t) does not depend on t. 


Definition 5.4.1. Let A be a positive random variable. If the counting process 
{N(t),t > 0}, given that A = X, is a Poisson process with rate A, then the 
stochastic process {N(t),t > 0} is called a conditional (or mixed) Poisson 
process. 


Proposition 5.4.1. The conditional Poisson process {N(t),t > O} has sta- 
tionary, but not independent increments. 


Proof. Consider the case when A is a discrete random variable whose possible 
values are in the set {1,2,...}. We have 


Me 


PIN(T +t) ~ N(r) =n] = PIN(r +t) —N(r) =n] A= k] = pa(k) 


k 


lI 
pt 


em ey" palk) (5.121) 


— 


Me 


k 


I 
bm 


Since N(0) = 0, for any value of A, we conclude that 
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P(N(7 +t) — N(r) =n] = P[N(t) =n] for all r,t > 0 and n=0,1,... 
(5.122) 


Thus, the increments of the process {N(t),t > 0} are stationary. 
Next, using the formula (5.121) above, we may write that 


PIN(t)=n|A=j]P[A=J] 
PIN(t) =n] 
-GO npa) 
Jae "onla 
a E A pal) 
S3 (5.123) 


Therefore, the number of events that occurred in the interval [0,t] gives us 
some information about the probability that the random variable A took on 
the value j. Now, the larger j is, the larger is the expected number of events 
that will occur in the interval (t,t +7], where T > 0. Consequently, we must 
conclude that the increments of the process {N (t), t > 0} are not independent. 
0O 


PIA = j | N(t) = n] = 


Remarks. i) In the case of the homogeneous Poisson process, with rate A > 0, 
we have that p,(A) = 1, so that 


PA= ALNG) =n] = <P form=O0,lie. (59.124) 


ii) The proposition above implies that a conditional Poisson process is not a 
Poisson process unless A is a constant. 


iii) If the parameter À of the homogeneous Poisson process {N(t),t > 0} is 
unknown, we can estimate it by taking observations of the random variable 
N(t), for an arbitrary t > 0. Suppose that we collected n (independent) ob- 
servations of N(t). Let X be the average number of events that occurred in 
the interval f0, t}. That is, 


Kea Deke Xe (5.125) 
n 


where Xx is the kth observation of N (t). The best estimator of the parameter 
A is given by 
(5.126) 


Thus, the larger the number of events that occurred in the interval [0,t] is, 
the larger the estimated value of À is. However, once this parameter has been 
estimated, we start anew and we assume that the increments are independent. 
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iv) When A is a continuous random variable, we find that 


e~ AEN" F(A) 


fanaa |n) = Pour fal) de VA>0 (5.127) 
Given that 
E[N(t)| A] = At and V{[N(t) | A] = At (5.128) 
we calculate 
E(N(t)| = E[E[N(t) | Al] = tE[A] (5.129) 


and 


VIN(t)] = E[V[N(E) | Al] + V[ELN() | 41] 
= E{At] + V[At] = tE[A] + t?V[A] (5.130) 


Example 5.4.1. Suppose that the rate of a Poisson process is a continuous 
random variable A such that 
a on 
fala) = 53 ifA>1 


It can be shown that 
oO n 
PIN(t) >n] = / PIA > Ate AUP dA 
0 


Use this formula to calculate the probability that there will be more than two 
events during an arbitrary time unit. 


Solution. First, we calculate 
a. 1 \>1 
PAS N= ; ~g dt = 35 for À > 
We seek 
1 2 o0 1 
À 1 A? 1 _ 
2| = A ei es e*— d= J) ve aise 
P[N(1) > 2] / l-e 5 ant | ye > J e 5 


Doing the integral above by parts, we find that 


1 
PINO) > 2] = 5 (2—5e7 +. e7') = 1 — 2e7t ~ 0.2642 
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Example 5.4.2. If A has a geometric distribution, with parameter p € (0,1), 
then we deduce from the formula (5.121) that 





(0 0) CO 
aia PEN Pel ea ee 
PIN(t) =1] = 7t Dkr See rats = Pre (2) 7a a? 








Example 5.4.3 (The Pélya’ process). Suppose that A has a gamma distri- 
bution, with parameters a = k € N and £8 > 0. We have 

P(N(t) = n] 
OO 


PIN(t) =n | A= Ajfa(d) dà 
(r)-1 
=l 


|l 


i 
Som oO | 


ye At)” 


e (Be~P*) -——__— dA 


n! (k—1)! 

=e pk I eo (E+ B)Ayntk-1 gy 

n! (k— 1)! Jo 
pe. p" 1 

~ nl (k—D! (t+ AT 
E 1 

= “aE Di Ga pyre e+) 
pe. -i 1 

~ nl (k—1)! (+ BFF 


= ene ates for = 0, 1522: 





| ety" t*—-l de (with « = (t+ B)A) 
0 


(n+k-1)! 


7 


where 


p := d 
' t+ 
1 George Pólya, 1887-1985, was born in Hungary and died in the United States. 
He contributed to several domains of mathematics, including probability theory, 
number theory, mathematical physics, and complex analysis. 
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We say that N(t) has a negative binomial, or Pascal? distribution, with 
parameters k and p. Furthermore, the process {N(t),t > 0} is called a megane 
binomial process, or a Pólya process. 


Remark. There exist various ways of defining a negative binomial distribution. 
The version above generalizes the geometric distribution when we define it as 
being a random variable that counts the number of failures, in Bernoulli trials, 
before the first success, which can be checked by setting k = 1. Thus, here N(t) 
would correspond to the variable that counts the number of failures before the 
kth success. Note also that if k = 1, then A has an exponential distribution 
with parameter ĝ. 


Definition 5.4.2. Let {A(t),t > 0} be a stochastic process for which A(t) 
is nonnegative, for allt > 0. If, given that A(t) = X(t), for allt > 0, the 
counting process {N(t),t > 0} is a nonhomogeneous Poisson process with 
intensity function A(t), then {N(t),t > 0} is called a doubly stochastic 
Poisson process or a Cox? process. 


Remarks. i) If the random variable A(t) does not depend on t, we retrieve 
the conditional Poisson process. Moreover, if A(t) is a deterministic function 
A(t), then {N (t), t > 0} is a nonhomogeneous Poisson process, with intensity 
function A(t). The doubly stochastic Poisson process thus includes the ho- 
mogeneous, nonhomogeneous, and conditional Poisson processes as particular 
cases. 


ii) The doubly stochastic Poisson process is not a Poisson process unless the 
random variable A(t) is equal to the constant A > 0, for all t > 0. 


iii) The process {A(t),t > 0} is called the intensity process. 
We can write, for all tg > tı > 0, that 


N(t2) — N(t1) | {A(t) = A(t) Y t > 0} ~ Poi(m(t2) — m(t)) (5.131) 


where 
t 
m(t) = | A(s) ds (5.132) 
0 
It can also be shown that 


g 2 A(t) at)” 


PIN (t2)— N(ti) =k | {A(t),0 <t) <t < te j] = eft 40 dt 7 


(5.133) 
fork =0,1,.... 


2 See p. 135. 
3 Sir David Cox, professor at the University of Oxford, in England. 
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To simplify the writing, set 
N(t) | {A(s),0 < s < t} = N(t) | A(0,t) (5.134) 
We then have 
EIN (t) | A(O,t)] = [4 ds (5.135) 


so that 


E|N(t)] = E[E[N(t) | A(0, t)]) = Bl [a A(s) ds| = f E|A(s)| ds (5.136) 


Similarly, we have 


2 


EIN?(t) | A(0,t)] = / A(s) ds + ( / A(s) is) (5.137) 


paa) af fsomren 
ay [ Bu E (5.138) 


E(N?(t)] = E[E[N2(t) | A(0,t)]] = [ea \] ds + [ Ra(s,u) ds du 
(5.139) 


Since 


E 








we may write that 


Example 5.4.4. Suppose that {A(t),t > 0} is a homogeneous Poisson process, 
with rate \ > 0. Then 


E{A(t)} =At and Ra(s,u) = Amin{s, u} + A*su 


We calculate l ; 
t 

E{N(t)] = f As ds = A— 

0 2 


and 


E(N?(t)] = =r 4 +f fi [A min{s, u} + A*su] ds du 


fe 
=k 4 +f [ rmints u} ds du + = 


: t u t t4 
ATTA sds+ | uds du + d\?— 
2 0 j 4 
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t? t3 i 
eae Rls No ae RO 
2 ü 3 a 4 
It follows that 


2 B tt 12\* 2 g 
VIN()] =A7 +A + — (5) =\(5+5 


Example 5.4.5. If {A(t),t > 0} is a geometric Brownian motion, then A(t) has 
a lognormal distribution (see p. 186), with parameters pt and o7t. Its mean 
is given by 


E[A(t)| = exp{ ut + 50°} 


We then have 
E(N(t)| = fo {us + 1 2s} ds = Or t+ on — 1| 


if u+ 507 £0 (and E[N(t)] =t if u + 40? = 0). 


5.5 Filtered Poisson processes 


After having generalized the homogeneous Poisson process in various ways, 
we now generalize the compound Poisson process. 


Definition 5.5.1. Let X1, X2,... be random variables that are i.t.d. and in- 
dependent of the Poisson process {N (t), t > 0}, with rate A > 0. We say that 
the stochastic process {Y (t), t > 0} defined by 


N(t) 
Y(t) =Y w(t, Ie Xz) Vt>0 (Y(t) =0 if N(t)=0) (5.140) 
k=] 


where the T,’s are the arrival times of the events of the Poisson process and 
the function w(-,-,-) is called the response function, is a filtered Poisson 
process. 


Remarks. i) The compound Poisson process is the particular case obtained by 
setting w(t, Tk, Xk) = Xk. 


ii) We can say that the response function gives the value at time t of a signal 
that occurred at time Tę and for which the quantity Xx has been added to 
the filtered Poisson process. The random variable Y (t) is then the sum of the 
value at time t of every signal that occurred since the initial time 0. 
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iii) An application of this type of process is the following: the variable Y (t) 
represents the flow of a certain river at time t (since the beginning of the 
period considered, for example, since the beginning of the thawing period), 
the 7j,’s are the arrival times of the precipitation events (snow or rain), and 
Xj, is the quantity of precipitation observed at time Tọ and measured in inches 
of water or water equivalent of snow. In practice, the precipitation does not 
fall to the ground instantly. However, we must discretize the time variable, 
because the flow of rivers is not measured in a continuous way, but rather, in 
many cases, only once per day. Then, the index k represents the kth day since 
the initial time and X, is the quantity of precipitation observed on this kth 
day. A classic response function in this example is given by 


w(t, Th, Xp) = Xpe7"'-7)/¢ for t > Ty (5.141) 


where c is a positive constant that depends on each river and must be esti- 
mated. 


Suppose that we replace the random variable Tk by the deterministic vari- 
able s in the function w(t, Tk, Xp). Let C,,(@) be the characteristic function 
of the random variable w(t, s, Xx): 


C.,(0) (= Cy(,t,8)) := E Cee) (5.142) 


Making use of Proposition 5.1.6, we can show that the characteristic function 
of Y(t) is given by 


Cy (t)(@) = exp f-x +À : Cy (8) ds) (5.143) 


from which we obtain the following proposition. 
Proposition 5.5.1. If E[w7(t,s,X,)| < œ, then the mean and the variance 
of Y(t) are given by 
t 
EIY(t)) =) / nO AEE. (5.144) 
0 


and 


ViY(t)] =A [ E{w*(t,s,Xp)| ds (5.145) 


Remarks. i) When Xx is a continuous random variable, the mathematical 
expectations E|w”(t,s, X)|, for n = 1,2, are calculated as follows: 


OO 


Elw" (t, s, Xx) =| w” (t, 8,2) fx, (x) dx (5.146) 


—OO 
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ii) It can also be shown that 


min{t1,t2 } 
Cov[Y (t1), Y (t2)] = a | Eļw(t;, S, Xp )w(te, S, Xx)| ds Vti t220 
0 
(5.147) 


Example 5.5.1. Suppose that the random variables X; have an exponential 
distribution with parameter u in the formula (5.141). We can then write that 


Cy(0) := E Cen) =E [e -E [etta] 


where 
g(t, 8,c) = e7 &7°)/e 


We have 


E eos) on f eJ99(t,8,0)@ e7 HE | goles = oe 
0 u — jAg(t, 8,c) 


Next, we calculate 


: : t un — iOa(t g(t, s, 
[ cw) is = | H is= f u — JOg(t, s,¢) + J09(t, 8:0) a 
0 o L— jOg(t, s,c) 0 u — J@g(t, s,c) 
t " 
jOg(t, s,c) 
-iy f -Be 4 
0 u — 78g(t, s,c) 
Given that 


o 1 1 
2. -eltel Loa 
Salts 8,¢) =e OM e~ = — alt, 8,0) 


we may write that 


t A 
j0g(t, s,c) . a 
———_—_-__——~ ds = —c İn |u — 709(t, s,c = 
I u — jOg(t, s,c) [u — jg )lls=0 


= —cln(p — j0) + cln(u — j0e~*/¢) 
It follows that 


Cy (0) = exp {at + Alt — cln(u — j0) + cln(u — joet} 


(£ a 
—\ pe 


Using this formula, we find that 


EIY(t)| = x (1 — e=t/°) and VrO] = “5 (1 _ ea 
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Remarks. i) In this example, we have 
Elw”(t, s, Xp)] eae 5 [xgene] = e~nt—-s)/¢ p [XP] 


7 n(t-s)/e P(n+1) 
un 


We then deduce from the formulas (5.144) and (5.145) that 


ElY (t)| =a f El w(t, s, Xk) Jds=a f e ae g eL ds =~ (1 — ee) 
H H 


Loo fons 0 laea 


and 
t t 9 
VY (t)] = a f Elw?(t,s,X,)| ds = a f E A ds 
0 0 H 
z Ac —2t/c 
ah (1 e ) 
as above. 


ii) We can check that the covariance of the random variables Y (t) and Y (t+7) 
is given by 


Cov[Y (t), Y@+7)] = Tse re (1 — e72t/c) ViT20 


Finally, we can generalize the notion of filtered Poisson process by setting 


Nit) 
Y= DD W(t, Tk) Vt>O (and Y(t) =0 if N(t)=0) (5.148) 


where {W,(t, s),t > 0,5 > 0} is a stochastic process (with two time parame- 
ters). We assume that the processes {W} (t, s),t > 0,8 > 0} are independent 
and identically distributed and are also independent of the Poisson process 
{N(t),t > 0} (having rate A > 0). We say that {Y (t),t > 0} is a generalized 
filtered Poisson process. It can be shown that 


EIY(t)) =. / ' EW, s] ds and ViV(é)) =A / " EIW2(t, 8)) ds 
(5.149) 


5.6 Renewal processes 


An essential characteristic of the Poisson process, {N(t),t > 0}, is that the 
time between consecutive events is a random variable having an exponential 
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distribution with parameter A, regardless of the state in which the process 
is. Moreover, the random variables 7; denoting the time that {N(t),¢t > 0} 
spends in state 7, for i = 0,1,..., are independent. 

We also know that the Poisson process is a particular continuous-time 
Markov chain. Another way of generalizing the Poisson process is now to 
suppose that the nonnegative variables 7; are independent and identically 
distributed, so that they can have any distribution, whether discrete or con- 
tinuous. 


Definition 5.6.1. Let {N(t),t > 0} be a counting process, and let 7; be the 
random variable denoting the time that the process spends in state i, for 
i = 0,1,.... The process {N(t),t > 0} is called a renewal process if the 
nonnegative variables T9,71,... are independent and identically distributed. 


Remarks. i) It is sometimes possible (see Ex. 5.1.2) to transform into a Poisson 
process a renewal process for which the time spent in the states 0,1,... does 
not have an exponential distribution. 


ii) We say that a renewal has occurred every time an event of the counting 
process takes place. 


iii) Some authors suppose that the random variables 79,71,... are strictly 
positive. 


iv) We can generalize the definition above by supposing that the random vari- 
able To is independent of the other variables, 71, 72,... but does not necessarily 
have the same distribution as these variables. In this case, {N(t),t > 0} is 
called a modified or delayed renewal process. 


The time Thn of the nth renewal of the (continuous-time) stochastic process 
{N(t),t > 0}, defined by [see Eq. (5.32)] 


n-1 
TS a DEE (5.150) 
i=0 


satisfies the relation Tn < t = N (t) > n. By setting To = 0, we may write, as 
in Proposition 5.1.4, that 


NG) = max{n 20: Irs ¢} (5.151) 
Since 
P|N(t) = n| = PIN(t) > n) — PIN() > n+] (5.152) 
we can state the following proposition. 


Proposition 5.6.1. The probability mass function of the random variable 
N(t) can be obtained from the formula 


PIN(t) = n] = Pl, < t] — Pin < t (5.153) 
fort > 0 and forn=0,1,.... 
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In general, it is difficult to find the exact distribution function of the 
variable Tp. We have 


m—1 


Cr, (w) := E eon ef Gi Da a D 





n—1 

{| ai 

i=0 

n-l n— ił 

= [JE [e7] = [] Cl) (5.154) 
i=0 i=0 


When 7; is a continuous random variable, the characteristic function Cr, (w) 
is the Fourier transform of the probability density function fr, (t) of Ta. We 
can then write that 


ÍT, (t) a Fro (to) * Ia (tı) Kese frai (tn-1) (5.155) 
where to + ti +... + tn-1 = t. That is, the density function of Ta is the 
convolution product of the density functions of To, ... ,Tn—1- 


In some cases, we know the exact distribution of the random variable Th. 
For example, if 7; ~ Exp(A), then 


Tr ~ G(n, à) (5.156) 
Similarly. if r; ~ Poi(A), we have 
Tn ~ Poi(ndA) (5.157) 
If n is large enough, we deduce from the central limit theorem that 
Tn © N(np, no”) (5.158) 
where u := Efr;] and a? := Vfr;] Y i. 


Example 5.6.1. We can use the formula (5.153) to check that, for a Poisson 
process with rate A, we have 


PING) =a) = e7% At" for t > 0 and n = 0,1,... 
n! 
In this case, 


(An)? + 


co" 


t 
Teta = Prete I Ne? 
Q 


Let f 
In m ee" dx forn=0,1,... 
0 
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Integrating by parts, we obtain 








e n 
In = ~z” —— | +r- 
A |g A 
t 
n =A 
EE EE a E rs 
=A A): À i A" 
= . n! moki n! I 
~ £4 (n-k+1) eT ye” 
where i A 
ree 
lgz= f a dy = ——* 
0 A 


It follows that 


Anti 2 n! pakti et n! /1-e7% 
PIT a4 <ia al - pU USU ` 


n n—k+1 n i 
sleg tey. (At) Pian (At)! At 





Similarly, 





Thus, we indeed have 


PiN(t)=n)= ( — D aen) — k -= 3 


i=0 1=0 








aen) _ Qt)" w 


a 


When 7; is a discrete random variable, the probability P[7; = 0| may be 
strictly positive. That is, the time needed for a renewal to occur may be equal 
to zero. For example, if 7; ~ Poi(A), then P[7; = 0] = e~* > 0. However, if 
the nonnegative random variable 7; is not the constant 0, we may write that 
u > 0. If we assume that u < œ and a? < ov, the strong law of large numbers 
(see p. 32) then implies that 


n—1 
lim duizo Ti a 
n= n 


1=P = P| ki =n] = P| im T = 00] =1 


n= 00 TE CO 








(5.159) 


from which we deduce from the formula (5.151) that we can write 


P(N(t) = co] = 0 for all t < co (5.160) 
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because the random variable T, will eventually be larger than any finite t. 
That is, there cannot be an infinite number of renewals in a finite-time interval. 
However, given that Pit; = oo] = 0, we may conclude that 


P [lim N(t) = 00 =4 (5.161) 


=> OO 


Since, in most cases, it is very difficult to explicitly calculate the probability 
of the event {N(t) = n}, we must generally content ourselves with finding the 
mean of the random variable N(t). 


Definition 5.6.2. The function my(t) := E[N(t)| is called the renewal 
function (or mean-value function) of the renewal process {N(t),t > O}. 


Proposition 5.6.2. The renewal function my (t) can be calculated as follows: 


OO 
mun (t) a PIT, < (5.162) 


Proof. As N(t) is a random variable taking its values in the set N° = 
{0,1,...}, we may write (using the relation Ta < t <= N(t) > n) that 


iP[N(t) =i] = 3 iPIN(t) =i] = > X PIN(t) =i] 


T 

Me 

a 
© 
A 


g (5.163) 


Remarks. i) Since it is also difficult to calculate the probability P[Tn < t], in 
practice the proposition does not enable us very often to obtain the function 
my (t). Moreover, if we managed to calculate P{[N(t) = n], for all n, then it 
is perhaps simpler to find the mean of N(t) by directly using the definition of 
E[N (t)). 

ii) We will further discuss another technique that enables us, when the 7;’s 
are continuous random variables, to calculate my(t), namely by solving an 
integral equation. 


Example 5.6.2. Suppose that 7; has a Bernoulli distribution, with parameter 
p € (0,1). We then have 


n—1 
Tin za ` Be B(n, p) 
i=0 


272 5 Poisson Processes 


so that i 
t 
n 
Tose > (pm p) 


where ft] denotes the integer part of t. Theoretically, we can calculate my (t) 
by finding the value of the sum 
[t] 
- x (1 = py -k 
1k=0 


Consider the particular case when p = 1/2. We have 


S(t,1/2) = S fa ry o 


n=i 


A 


S(t, p) := 


n 


where (7 )=O0ifk>n. 


We find, making use of a mathematical software package, that S(0,1/2) =1, 
S(1,1/2) = 3, $(2,1/2) = 5, S(3,1/2) = 7, etc., from which we deduce the 
formula 

S(t,1/2) = 2{t]+1 for allt >0 


Now, let t =r € {0,1,...}. Given that 7; = 0 or 1, for all 7, we may write 
that N(r) > r. We find that 


r with probability p’t} 


ji 
r +1 with probability (" H )rna — p) 
1 
N(r) = 


ie 
r +2 with probability (" 5 ) ptt — p} 


In general, we have 
PIN(r) =r + k] = Coa —p)* fork=0,1,... 


Remark. We have that PIN (r) = r] = p™t+, and not p”, because {N (r) =r} if 
and only if (iff) the first r + 1 renewals each take one time unit. In particular, 
N(0) = 0 iff 7) = 1. Similarly, N(1) = 1 iff m = 1 and T, = 1, etc. 

We may write that X := N(r) + 1 has a negative binomial distribution 
(see p. 135) with parameters r + 1 and p. Now, the mean of this distribution 
is given by (r + 1)/p, from which we deduce that 
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E[N(r)) = 4 Ue 





-1 => E|N(t)) = 


Note that if p = 1/2, then we retrieve the formula S(t, 1/2) = 2[t]+ 1 obtained 
previously. 


Suppose that the 7;’s are continuous random variables. We then deduce 
from Eq. (5.162) that 


“my (t = ft, (5.164) 


from which, taking the Laplace transform of both members of the preceding 
equation, we obtain 


my (@) = a e“dmy(t) = ` a e% fr, (t) dt (5.165) 
n=l 
<> my(a)= ` Mr, (a) (5.166) 
n=i 


where Mr, is the moment-generating function of the random variable Tn 
(see p. 19) and a is a negative constant. Since the 7;’s are independent and 
identically distributed random variables, we may write that 


pe Ta (a) = [M,,(a)]” (5.167) 


i=0 


The parameter a being negative, we have that M,,(a) € (0,1). It follows that 


= doe ta (5.168) 
= of 1 — M,,(a@) 
so that 
my (a) 
Moy 5.169 
o(a) 1+ mila) ( ) 


By the uniqueness of the Laplace transform (and of the inverse Laplace trans- 
form), we deduce from the last equation that to each renewal function my (t) 
corresponds a different density function fr (t). Similarly, Eq. (5.168) implies 
that the density function f,,(t) uniquely determines the function my (t). 
Moreover, it can be shown that these results hold true, whether the ran- 
dom variables 7; are discrete or continuous. We can thus state the following 
proposition. 
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Proposition 5.6.3. There is a one-to-one correspondence between the distri- 
bution function of the random variables t; and the renewal function my (t). 


_ Since my(t) = At in the case of the Poisson process, we have the following 
corollary. 


Corollary 5.6.1. The Poisson process is the only renewal process having a 
linear renewal function. 


Remark. The Poisson process is also the only Markovian renewal process. 


In the case when the random variables 7; are continuous, we have 


mv (t) = BING] = EENG | ral] = f ” BIN(t) | To =T]fro(7) d7 


(5.170) 
Moreover, the relation 
Ty >t 4 N(t)=0 (5.171) 
implies that 
t 
my(t) = / E\N(t) | To = T} fa (T) dT (5.172) 
0 


Finally, we may write (the 7;’s being i.i.d. random variables) that 
E|N(t)| 7m =7]=1+E[N(t-r)]=1+my(t—-r) forO<7<t (5.173) 
It follows that 
t 
n= / E mites (5.174) 
0 
t 
<=> my(t) = Falt) +f my(t—T)fp (7) dr fort >0 (5.175) 
0 


Definition 5.6.3. Equation (5.175) is called the renewal equation (for the 
renewal function) of the process {N(t),t > 0}. 


To explicitly calculate the renewal function, it is often easier to solve the 
integral equation (5.175). Setting s = t — 7, we can rewrite it as follows: 


t 
mn(t) = Fan (t) +j mn(s)fr,(t-—s) ds fort>0 (5.176) 
0 
Next, differentiating both members of the preceding equation, we obtain 


my (t) = fro (t) + muy (t) fr (0) + / my(s)f',(t— 9) ds (5.177) 
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When f,,(t — s) is a polynomial function, we can differentiate repeatedly 
the members of this equation, until we obtain a differential equation for the 
function my (t). We have, in particular, the boundary condition my (0) = 0. 


Remark. We can also sometimes obtain a differential equation for my (t) even 
if f,,(¢ — s) is not a polynomial function. 


Example 5.6.8. Let 
fo ae fO<t<1 


We have [see Eq. (5.177)]: 
t 
my (t) = 4t? + my(t) x 0+ / my (s)12(t — s} ds 
0 
t 
= 473 + mn(s)12(t—s)* ds forO<t<1 
0 
Note that this equation implies that m‘,(0) = 0. We differentiate once more: 
t 
mix, (t) = 12° + f my (s)24(t — s) ds 
0 | 
from which we deduce that m‘4,(0) = 0. Next, we have 
t 
my (t) = 24t + 24 f my(s) ds 
0 
so that m% (0) = 0. Finally, we obtain the ordinary differential equation 
m) (t) = 24 + 24my (t) 
N aa N 


The general solution of this equation is given by 


mn (t) = —1 + cı cos kt + cg sin kt + cge" + cae 


where k := (24)!/4 and c1, ..., c4 are constants. The particular solution that 
satisfies the conditions my (0) = m4 (0) = m% (0) = mý (0) = 0 is 


1 ] 
my(t) =—-1+ 5 cos kt + i (e"t + ant) forO<t<1 
Note that the solution above is only valid for t € [0,1]. Moreover, since the 
random variable 79 takes its values in the interval [0, 1], the mean my (1) must 


be greater than 1. We calculate 


1 1 
my(1)=-1 +3 cosk+ 7 (e° +e") ~ 1.014 
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Equation (5.175) is a particular case of the general renewal equation 
OO 
g(t) = h(t) + f g(t —T)dF,,(7) fort >0 (5.178) 
0 


where h(t) and F,,(t) are functions defined for all t > 0. The following propo- 
sition can be proved. 


Proposition 5.6.4. The function g(t) defined by 
t 
g(t) = h(t) + J h(t—7)dmy(r) fort>0 (5.179) 
0 
is a solution of the renewal equation (5.178). 


Remarks. i) Equation (5.175) is obtained from (5.178) by taking g(t) = my (t) 
and A(t) = F,,(t) and by noticing that my(t) = 0 if t < 0. If h(t) is a function 
bounded on finite intervals, then g(t) is bounded on finite intervals as well. 
Moreover, g(t) is the unique solution of (5.179) with this property. Note that 
the distribution function F,,(t) (= A(t)) is evidently a bounded function, 
because 0 < F,,(t) <1 Vti>0. 

ii) We can replace dmy(r) by m‘,(7) dr in the integral above. 


Corollary 5.6.2. If To is a continuous random variable, then the second mo- 
ment of N(t) about the origin, E[N?(t)], is given by 


E(N*(t)] = mn(t) +2 7 mn(t—T)dmy(t) fort >0 (5.180) 
0 
Proof. We may write that 
E(N?(t)] ap EIN? (t) | To = T] fm (T) dr (5.181) 
t t 
-| EJN? (t) | To = T] fn (T) dr = f E[(1+N(t—1))*] fro(T) dr 
0 0 
e J {1 + 2E[N(t —7)] + EIN?(t — 7))} fr (7) dr 
=Fn(t) +2 | mult- Tfal) dr | EIN E- Tfr) dr 
0 0 
Making use of Eq. (5.175), we obtain 


E[N? (H) = 2mn(t) — Fa (t) + [ EIN? (t —7)]fro(r) dr (5.182) 
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This equation is of the form of that in (5.178), with g(t) = E[N?(t)] and 
h(t) = 2my (t) — Fa (t) (since g(t) = 0 if t < 0). It follows, by Proposition 
5.6.4, that 
t 
E(N?(t)] = 2my(t) — Feo (t) + J Rmy(t—T)— F,,(t — T) dmy (T) 
0 
(5.183) 
We have 


I 


t t 
f Felt—1) amy(r) = Falt- rmn) | Falt -rmui dr 
0 0 


i 


t 
0 +j fro(T)mn(t — T) dt 
0 
SEO my(t) — Fn lt) (5.184) 
from which we obtain Eq. (5.180). O 
Example 5.6.4. When {N(t),t > 0} is a Poisson process with rate A, we have 
E(N? (t)] = VIN(t)] + (EIND? = At + (A0)? 


which is indeed equal to 


t t 
my (t) + 2 | mnu(t—7) dmy(r) = At + 2 | A(t ~ T)A dr 
0 0 


ofa C 
= M + 2A? (2-7) 


falt) =1 if0<t<1 


Example 5.6.5. When 


we find that 
E(N(t)}=e’~1 forO0<t<1 


It follows that 
t 
E(N?(t)] = ef —1+ 2 | (e7 — 1) e” dr 
0 


t 

=e -142/ (é —e7) dr=e' -14+2 |te’ — (e — 1) 
0 

=e'(2t-—1)+1 for0<t<1 


As in Example 5.6.3, the fact that 0 < Tọ < 1 implies that E[N*(1)] > 1, 
for k =1,2,.... We have 


E[N(1)}=e-1 and E[N*(1))=e+1 


278 5 Poisson Processes 


We know that the sum of two independent Poisson processes, with rates A, 
and Ag, is a Poisson process with rate A := A, +2 (see p. 234). The following 
proposition can be proved. 


Proposition 5.6.5. Let {Ni(t),t > 0} and {No(t),t > 0} be independent 
renewal processes. The process { N(t),t > 0} defined by 


N(t) = Ni(t)+ No(t) fort >0 (5.185) 


is a renewal process if and only if {N;(t),t > 0} is a Poisson process, for 
ee 


Finally, the following result can be proved as well. 


Proposition 5.6.6. Let {N(t),t > 0} be a renewal process. The kth moment 
of N(t) about the origin (exists and) is finite, for all k € {1,2,3,...}: 


E(N*(t)| < for allt < œ (5.186) 


Remark. We deduce from the proposition that my(t) < oo Vt < oo. Thus, not 
only can the number of renewals in an interval of finite length not be infinite 
(see p. 270), but the mathematical expectation of the number of renewals in 
this interval cannot be infinite either. 


5.6.1 Limit theorems 


Note first that Tyna) designates the time instant of the last renewal that 
occurred before or at time t. Similarly, Tyy(4)41 denotes the time instant of 
the first renewal after time t. 

We will show that the average number of renewals per time unit, namely 
N(t)/t, tends to 1/p as t — œ, where u := Efri] Y i is assumed to be finite. 
The constant A := 1/p is called the rate of the process. 


Proposition 5.6.7. We have, with probability 1: 


a >À ast-@w (5.187) 


where A = 1/p > 0. 


Proof. We may write that 


Tn (t) STS TN (5.188) 


Ini) o t - INO+H a 
Na) ao < way FNO>% (5.189) 





Now, by the strong law of large numbers (since E[|7;|] = E[7;] < 00), we have 
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N(t)-1 


In Tj 
(t) >D NO — u as N(t)—> co (5.190) 
i=0 


N(t) 





with probability 1. Furthermore, because N(t) — œœ as t -> 00, we may write 
that (with probability 1) 








TN) 
= .191 
too N(t) (5.191) 
Finally, given that 
Fue. Eves (i 4 xis) (5.192) 
N(t) N(t)+1 (t) 
we deduce that 
t 1 Ni) 1 
< lim —— < — < lim —— < — 5.193 
Po NG eee eo one 


with probability 1. O 


The next theorem, which states that the average expected number of re- 
newals per time unit also converges to X, is not a direct consequence of the pre- 
ceding proposition, because the fact that a sequence 2, Ce ee of random vari- 
ables converges to a constant c does not imply that the sequence {E Xn] } n1 


converges to c. 


Theorem 5.6.1 (Elementary renewal theorem). If the mean Ejro] is fi- 
nite, then we have 


fom ENO) 


t—00 t 


2A (5.194) 


Remark. i) Actually, if u = oo, the preceding result is still valid (by setting 
à := 1/p = 0). Similarly, Proposition 5.6.7 is valid when u = œœ as well. 

ii) There is no mention of probability in the theorem, because E[N (t)] is a 
deterministic function of t (while N(t) is a random variable). 


The following two theorems can also be shown. 
Theorem 5.6.2. If u := E[r] < œ and o? := V [ro] < 00, then 
an VINO 


t—00 t 


= 0°)” (5.195) 


Theorem 5.6.3 (Central limit theorem for renewal processes). If 
E{r@] is finite, then we may write that 
t to? 


N(t)=N (+ Z) = N (t,o? A't) ift is large enough (5.196) 
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Example 5.6.6. lf {N(t),t > 0} is a Poisson process with rate Ag, then 7 ~ 
Exp(Ao), so that u := E[79] = 1/Ap and o? := V[79] = 1/A2. We have indeed 


N(t) ~ Poi(Aot) ~ N(ot, Aot) = N(At, Ap 7A°t) 
where À := 1/2 = Xo. 


The proposition that follows can be used to show Theorem 5.6.1. 
Proposition 5.6.8. We have 
E|Tn(t)41] = Efo {EIN (t) + 1} (5.197) 


Proof. By conditioning on the value taken by the random variable 7, which 
is assumed to be continuous, we obtain 


ETna] = f E|Tnaj+1 | To = T| fro (7) dT 
t oe) 
= | {r + E|Tya-r)+1]} falt) dT +j Fit) ar 
0 t 
OO t 
= f tho(r) drt [Ett a-nsilfo(r) dr 
0 0 


t 
= Elo) + | ElTva-nsslfno(t) dr (5.198) 


This equation is the particular case of the renewal equation (5.178), which is 
obtained with g(t) = E[Ty (41) and h(t) = E[79]. Dividing both members of 
the equation by E{79] and subtracting the constant 1, we find that 


ElTva+i] [ E\Ty(e-1)+1] 


sO = Bg fot) dr (5.199) 


t 


= f it-1) + fal) dr = Falt) + f g- frl) dr 
0 l 0 


Now, this equation is the same as Eq. (5.175). Therefore, by the uniqueness 
of the solution, we may write that 


El Tw t)+1] 


oN Ef[ro] 


-1 <4 EļlTunæ] = Elrollmn(t) +1] 0 
(5.200) 


Definition 5.6.4. Lett be a fixed time instant. The random variable 
A(t) := t—Twyqy (5.201) 
is called the age of the renewal process at time t, while 
D(t) := Tne —t (5.202) 


is the remaining or excess lifetime of the process at time t (see Fig. 5.3). 
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A(t) DO 
T NÀ f T N+ 


Fig. 5.3. Age and remaining lifetime of a renewal process. 


By proceeding as above, we can prove the following proposition. 


Proposition 5.6.9. The distribution function of D(t) is given by 


Ł 
Fow(r) = Fa(t+r)— | A S E E (5.203) 
Q 
forr >0. In the case of A(t), we find that 


P[A(t) > a] = 1 — Fa (t) + fu —F,,(t-—7T)] dmn(r) ifO<a<t 
i (5.204) 


and P[A(t) >al=Oifa>t. 


When ¢ is large enough and 79 is a continuous random variable, we find 
that 


1 
foan= Eire)! — F,,(r)| for r > 0 (5.205) 
In fact, this formula for the probability density function of D(t) is exact if the 
renewal process {N (t), t > 0} is stationary. 


Suppose now that at the moment of the nth renewal of the process 
{N(t),t > 0}, we receive, or we have accumulated, a reward Rn (which may, 
actually, be a cost). Suppose also that the rewards {Rn an are independent 
and identically distributed random variables. However, in general, R, will de- 
pend on 7,1, that is, the length of the nth renewal period, called a cycle. Let 
R(t) be the total reward received in the interval [0, t]. That is, 


N(t) 
ROSS Ry, (R(t) = 0 if N(t) =0) (5.206) 
n=1 


We will show that the average reward received by time unit, in the limit, is 
equal to the average reward received during a cycle, divided by the average 
length of a cycle. 


Proposition 5.6.10. If E{Ri| < œ and E[To] < œ, then we have 


_ R(t) ELRy)) 
P| jim = = aaa Si (5.207) 
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Proof. It suffices to use the strong law of large numbers and Proposition 
5.6.7. Since N(t) — 00 as t — 00, we may write 


N(t) | 
lim L Ni x G = E[R]| =1 (5.208) 


t= 


from which we have 


IW NO R NG 
lim Ae = lim — ee lim eae lim co 


t—00 too t too} N(t) |t-oo| t 
= E{R,| x —— (5.209) 
with probability 1. 0 
Remarks. i) It can also be shown that 
; E\R(t)| ER] K 
Ss T .210 
r O S 77 e 


ii) If the reward R, may take positive and negative values, then we must 
assume that E||Rı|] is finite. 


With the help of the preceding proposition, we will prove the following 
result. 


Proposition 5.6.11. If 7 is a continuous random variable, then we have, 
with probability 1: 


jim n = jim i 9 EI o] ( ) 





Proof. Suppose that the reward received at time t is given by A(t). Then, we 


may write that 
TO 2 E 2 
E[Ri] = E | f T ir > r| = 1) (5.212) 
0 





2 2 


To obtain the other result, we simply have to suppose that the reward received 
at time t is rather given by D(t), and then the variable 7 is replaced by 7) — 7 
in the integral. O | 


Remark. Note that we calculated the average value of the age (or of the re- 
maining lifetime) of the process over a long period, which is a temporal mean, 
and not a mathematical expectation. i Eftr] < oo, then it can also be shown 
that 


Efré) 


jim E[A(t)] = lim E[D()] = Ein] (5.213) 
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Example 5.6.7. lf {N(t),t > 0} is a Poisson process, then, making use of the 
fact that it has independent and stationary increments, we can assert that 
D(t) has an exponential distribution with parameter A. Thus, we have 


1 
E\D(t)| = = 
[De] = 5 
Moreover, To ~ Exp(A), too, so that 
t 2 
iin Jo DCT) dr _ xz + (x) _i 
t>o t 2/X A 


So, in this case, the temporal mean and the mathematical expectation of DN) 
are equal, for all t > 0. 


Example 5.6.8. iC Let {N(t),t > 0} be a counting process for which the time 
between two events has a U[0, S] distribution, where 


ge 1 with probability 1/2 
~ | 2 with probability 1/2 


Explain why {N(t),t > 0} is not a renewal process. 
(b) In part (a), suppose that S is rather a random variable whose value is 
determined at time t = 0 and after each event by tossing a fair coin. More 
precisely, we have 
ge l 1 if “tails” is obtained 
~ | 2 if “heads” is obtained 


In this case, the stochastic process {N(t),t > 0} is a renewal process. 

(i) Calculate my (t), for 0 <t <1. 

(ii) Suppose that we receive a reward equal to $1 when the length of a 
cycle is greater than 1 (and $0 otherwise). Calculate the average reward per 
time unit over a long period. 


Solution. (a) {N(t),t > 0} is not a renewal process, because the times be- 
tween the successive events are not independent random variables. Indeed, 
the smaller To is, the larger the probability that S = 1 is. So, the 7’s depend 
on To, for all k > 1. 


(b) (i) Let 7 be the length of a cycle. We have 
F(t) = Pir < t] =Pi[r <t| S =1] PIS = 1|4P[r < t| S$ =2|P[S = 2] 
ee 
1/2 
That is, 


2 


It follows that the renewal equation is 


1 
F(t) =- 5 (t+5) = Z for0<t<1 
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3t f ee E 
= — aes ae = — Aaa d 
my (t) 1 +| mit x) 7 dx 4 Eg if mn(y) dy 


3 3 
j — — — 
We deduce from the initial condition my (0) = 0 and from the formula (3.233), 
p. 130, that 


t 
my (t) = e®!/4 (o+ | e0/4e iy) =e /4_1 for0<t<1 
0 
(ii) We have 


Em] = Elo | $= 1] PIS = 1] +B | $= 21PIS = 2 = 5 (5 +1) => 


1/2 


and 


1 
E(Ri] = E[R, | S = 1) PIS = 1] + E[R |S =2) P[S == 7 
/2 
0 1 


Therefore, the average reward per time unit (over a long period) is given by 


(1/4)/(3/4) = 1/8. 


5.6.2 Regenerative processes 


We know that the Poisson process, {N(t),¢t > 0}, starts anew, probabilisti- 
cally, from any time instant, because it has independent and stationary incre- 
ments. However, since it is a counting process, once the first event occurred, 
N(t) will never be equal to 0 again. We are interested, in this subsection, 
in processes that are certain to eventually return to their initial state and 
that, at the moment of this return, start afresh probabilistically. This type of 
stochastic process is said to be regenerative. 


Definition 5.6.5. Let To be the time that the discrete-state stochastic process 
{X(t),t > 0} spends in the initial state X(0). Suppose that 

Pit > 72 X(t) =X (0)| = 1, 

ii) the processes {X(t) — X(0),t > 0} and {Y(t),t > 0}, where 


Y(t) := X(t +T) — X(T) (5.214) 


and T, is the time of first return to the initial state, are identically distributed, 
iii) the stochastic process {Y (t),t > 0} is independent of the process { X(t), 
0<t<Ti}. 

The process {X(t),t > 0} is then called a regenerative process. 
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Remarks. i) Let Ta be the time of the nth return to the initial state, for 
n = 1,2,.... We assume that PT} < oo] = 1. It then follows that the time 
instants T2, T3,... must also (exist and) be finite with probability 1. 

ii) A regenerative process may be a discrete-time process {Xn, n = 0,1,...} 
as well. In this case, To is the time that the process takes to make a transition 
from the initial state to an arbitrary state, which may be the initial state (for 
example, if {X,,n = 0,1,...} is a Markov chain). 


iii) The cycles of the regenerative process are the processes 


(Colt) Ostan) and {Osi kt fag} for m= Ow 


(5.215) 
These stochastic processes are independent and identically distributed, for 
n=0,1,.... Similarly, the random variables Xo := T; and Xn := Ty41 — Thn, 
for n= 1,2,..., are ii.d. 


iv) According to the definition above, a renewal process is not regenerative, 
because it is a particular counting process. However, we can associate a re- 
newal process {N(t),t > 0} with a regenerative process {X (t), t > 0}, by 
setting that N (t) is the number of times that {X(t),t > 0} came back to 
the initial state and started anew in the interval [0,t]. That is, N(t) is the 
number of cycles completed in this interval. Conversely, we can define a re- 
generative process {X(t),t > 0} from a renewal process {N(t),t > 0} (for 
which N(0) = 0) as follows: 


N(t) if0<t<T* 
NORTT 
a nit) — ok if Te, <t < Te, Dey) 


for some k € {2,3,...}, where Tř, T5,... are the arrival times of the renewals. 
Note that with k = 1, we would have that X(t) = 0. Moreover, here T = Tk. 


Example 5.6.9. The queueing model M/M/1, described in Example 3.1.6 (and 
which will be studied in detail in Chapter 6), is an example of a regenerative 
process. In this model, X(t) designates the number of persons in a system at 
time t. Thus, if X(0) = 0, then we are certain that the system will eventually 
be empty again and that, from that point on, everything will start anew. 


Example 5.6.10. Let {Xy,n = 0,1,...} be a discrete-time Markov chain, 
whose state space is the set {0,1} and whose one-step transition probabil- 


ity matrix is given by 
11/212 
cabins 


Suppose that Xo = 0. Since the chain is irreducible and has a finite number 
of states, we may assert that it is recurrent. Therefore, the probability that 
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the process will visit the initial state 0 again is equal to 1. It follows that this 
Markov chain is a regenerative process. Suppose that every transition of the 
chain takes one time unit. Then, we may write that P[T; = 1] = P[T; = 2] = 
1/2. Finally, let N(0) = 0 and N(t) be the number of visits to state 0 in the 
interval (0, ¢], for t > 0. The process {N(t),t > 0} is then a renewal process. 


Proposition 5.6.12. If E|T;| < œ, then the proportion ty of time that a 
regenerative process spends in state k is given, in the limit, by 


E|time spent in state k in the interval [0,T,]| 


= ET] 


(5.217) 
Proof. Suppose that we receive an instantaneous reward of $1 per time unit 
when the process {X (t), t > 0} is in state k, and $0 otherwise. Then the total 
reward received in the interval [0,t] is given by 


t ‘ 
R(t) =| Ls x(r)=k} dT (5.218) 
0 
where I;x(7)=4} is the indicator variable of the event {X(r) = k} [see 
Eq. (3.43)|. If E[T;] < oo, Proposition 5.6.10 implies that 
R 
ie e E (5.219) 





t=œ t ED] 


(with probability 1), where R; is the time spent by the process in state k 
in the interval [0,7]. Now, limz_,.o R(t)/t is the proportion of time that the 
process spends in state k. O 


Remark. The proposition is valid for any type of random variable Tı. When 
T; is a continuous variable, it can also be shown that 


dim P|X(t) = k| = Tk (5.220) 


That is, 7, is the limiting probability that the process will be in state k at 
time ¢ as well. 


Example 5.6.11. In the case of the process {X(t),t > 0} defined by Eq. 
(5.216), we can directly write that m9 = 71 =... = nk-1 = 1/k, because 
the random variables Tř, Tž,... are independent and identically distributed. 


Example 5.6.12. In Example 5.6.10, the reward R, is equal to 1, because the 
Markov chain spends exactly one time unit in the initial state 0 per cycle. It 


follows that 
1 1 2 


Ko = zr = r = — 


ET] 3/2 3 
We can check, using Theorem 3.2.1, that the proportion 7 is indeed equal 
to 2/3. Note that we may apply this theorem to obtain the proportion 7o, 
whether the chain is periodic or not. Here, the chain is ergodic, since poo = 
1/2 > 0. Then, 7 is also the limiting probability that the process will be in 
state 0 after a very large number of transitions. 
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Example 5.6.13. We consider a discrete-time Markov chain {X,,n = 0,1,...} 
with state space {0,1,2} and transition matrix P given by 


01/21/2 
P=/10 0 
01/21/2 


Suppose that the initial state is 0 and that when the process enters state 2, it 
remains there during a random time having mean u; for i = 0,1, 2, indepen- 
dently from one visit to another. Calculate the proportion of time that the 
process spends in state 0 over a long period. 


Solution. The Markov chain is recurrent, because it is irreducible and has a 
finite number of states. Then the process {X,,n = 0,1,...} is regenerative. 
Let T; be the time of first return to state 0 and N be the number of transitions 
needed to return to 0. We have 


ET; | N =n] = uo + u + (n —2)u2 for n= 2,3,... 


Since P(N = n] = (1/2)""1, for n = 2,3,..., it follows that 


ET] = EIET | N]| = Yb + pı + (n — 2)ptg](1/2)"* 
Finally, we can write that i 
Ye = = xe — 1)(1/2)” = E[Geom(1/2)] -1=1 
so ae i 


E[T,] = uo + H1 + pe 


The proportion of time that the process spends in state 0 over a long period 
is thus given by uo/(xo + Hı + u2). 

Remark. Since the matrix P is finite and doubly stochastic, we deduce from 
Proposition 3.2.6 (the Markov chain being irreducible and aperiodic) that the 
limiting probabilities 7, exist and are given by 1/3, for k = 0,1,2. Hence, we 
deduce that the proportion of time requested is indeed equal to uo/(Ho + “a + 
H2). 


An important particular case of a regenerative process is the one for which 
the state space of the process {X (t), t > 0} contains only two elements, which 
will be denoted by 0 and 1. For example, X(t) = 0 if some machine is down, 
and X(t) = 1 if it is operating at time t. Suppose that the machine is brand- 
new at the initial time 0, so that X (0) = 1, and that the time during which 
the machine operates, before breaking down, is a random variable S1. Next, 
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the repairman takes a random time U; to set the machine going again, and 
then the system starts afresh. Let 


Ties Sat Un forn = |, 2,33: (5.221) 


where Sn is the operating time of the machine between the (n — 1)st and 
the nth breakdown, and U,, is the repair time of the nth breakdown, for 
n = 2,3,.... We assume that the random variables {Sn} -1 are iid. and 
that so are the r.v.s Un) ar However, the r.v.s Sp and Un may be dependent. 
This type of stochastic process is often called an alternating renewal process. 


Remark. Let N(t) be the number of times that the machine has been repaired 
in the interval [0,t]. The process {N(t),t > 0} is a renewal process and the 
T,,’s are the arrival times of the renewals. However, the process {X (t), t > 0} 
defined above is not a renewal process (according to our definition), because 
it is not a counting process. 


By using the renewal equation, we can show the following proposition. 
Proposition 5.6.13. Let m(t) be the probability that X(t) = 1. We have 
t 
m= Rs | ketr tenon (5.222) 
0 


Next, we deduce directly from Proposition 5.6.12 that 
E[S] E[S] 


E a RE ee ee 5.223 
a "0 pat EU] BT] Sa 

Moreover, if Tı is a continuous random variable, then 
j = 224 
jim m(t) = ™ (5.224) 

Finally, let {N (t), t > 0} be a renewal process and 

(Lif A(t) <a | 

_ 5.225 
AW) : ae: | (p22) 


where a > 0 is a constant and A(t) is defined in Eq. (5.201). The process 
{X(t),t > 0} is then an alternating renewal process, and we deduce from the 
formula (5.223) that 


_ ElS;]  E|min{T*,a}] 


= Ein] > EIT ee) 





Wy 


where Ty is the arrival time of the first renewal of the stochastic process 
{N(t),t > O}. 
Remark. If TË < a, then we set U, = 0. 
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In the continuous case, we have 


E\min{Ty, a} = > min{t, a} fr» (t)dt = f tfr» (t)dt +f afr«(t)dt 
(5.227) 


We calculate 
a a a 
| tfrs (t)dt = tFr» Gl — / Fr: (t)dt = aFr»(a) — f Fr» (t)dt (5.228) 
0 0 0 
from which we obtain the formula 


E|min{ Ty, a}) = aF px (a) — [ Fr» (t)dt + aP[TÝ > a) = a — [ Fr» (t)dt 
(5.229) 


We can thus write that 


aA (5.230) 


Ni = 


Remark. We obtain exactly the same formula as above for m, if we replace 
A(t) by D(t) (defined in (5.202)) in (5.225). 


Example 5.6.14. If T¥ ~ Exp(A), then we have that PITY > t] = e~**. It 


follows that i 
fi en dt — Xa 


7 


y= 
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Section 5.1 


Question no. 1 
Let {N (t), t > 0} be a Poisson process with rate A. We define the stochastic 
process {X(t),0 <t < c} by 


X(t) = N(t) - ~N(e) for0<t<c 


where c > 0 is a constant. 

(a) Calculate the mean of X(t). 

(b) Calculate the autocovariance function of the process {X(t),0 < t < c}. 
(c) Is the process {X(t),0 < t < c} wide-sense stationary? Justify. 
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Question no. 2 

We suppose that customers arrive at a bank counter according to a Poisson 
process with rate A = 10 per hour. Let N(t) be the number of customers in 
the interval {0, t]. 
(a) What is the probability that no customers arrive over a 15-minute time 
period? 
(b) Knowing that eight customers arrive during a given hour, what is the 
probability that at most two customers will arrive over the following hour? 
(c) Given that a customer arrived during a certain 15-minute time period, 
what is the probability that he arrived during the first 5 minutes of the period 
considered? 
(d) Let X(t) := N(t)/t, for t > 0. Calculate the autocovariance function, 
C'x(t1,t2), of the stochastic process {X(t),t > 0}, for t1,t2 > 0. 


Question no. 3 
The stochastic process {X(t),¢ > 0} is defined by 


N (t+ 6?) — N(t) 
ô 


where {N(t),t > 0} is a Poisson process with rate À > 0, and 6 > 0 is a 
constant. 
(a) Is the process {X (t),t > 0} a Poisson process? Justify. 
(b) Calculate the mean of X(t). 
(c) Calculate the autocovariance function of the process {X(t),t > 0}, for 
ti =l, to = 2, and 6 = 1. 
(d) Let Zn := N(n), for n =0,1,2,.... 

(i) The stochastic process {Z,,,n = 0,1,...} is a Markov chain. Justify 
this assertion. 

(ii) Calculate p; j, for i,j € {0,1,2,...}. 


X(t) = fort > 0 


Question no. 4 

Let N(t) be the number of failures of a computer system in the interval 
|0, t]. We suppose that {N (t), t > 0} is a Poisson process with rate A = 1 per 
week. 
(a) Calculate the probability that 

(i) the system operates without failure during two consecutive weeks, 

(ii) the system will have exactly two failures during a given week, knowing 
that it operated without failure during the previous two weeks, 

(iii) less than two weeks elapse before the third failure occurs. 


(b) Let 


Z(t):=e NY fort >0 
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Is the stochastic process {Z(t),t > 0} wide-sense stationary? Justify. 
Indication. We have that E[e~°*] = exp [a (e~* — 1)] if X ~ Poi(a). 


Question no. 5 

A man plays independent repetitions of the following game: at each repe- 
tition, he throws a dart onto a circular target. Suppose that the distance D 
between the impact point of the dart and the center of the target has a UJO, 30] 
distribution. If D < 5, the player wins $1; if 5 < D < 25, the player neither 
wins nor loses anything; if D > 25, the player loses $1. The player’s initial 
fortune is equal to $1, and he will stop playing when either he is ruined or his 
fortune reaches $3. Let X,, be the fortune of the player after n repetitions. 
Then the stochastic process {X,,n = 0,1,...} is a Markov chain. 


(a) Find the one-step transition probability matrix of the chain. 
(b) Calculate E[X3]. 


Suppose now that the man never stops playing, so that the state space of 
the Markov chain is the set {0,+1,+2,...}. Suppose also that the duration T 
(in seconds) of a repetition of the game has an exponential distribution with 
mean 30. Then the stochastic process {N(t),t > 0}, where N(¢) denotes the 
number of repetitions completed in the interval {0, t], is a Poisson process with 
rate À = 2 per minute. 


(c) Calculate the probability that the player will have completed at least three 
repetitions in less than two minutes (from the initial time). 


(d) Calculate (approximately) the probability P[N(25) < 50]. 


Question no. 6 

Let N(t) be the number of telephone calls received at an exchange in the 
interval [0, t]. We suppose that {N(t),t > 0} is a Poisson process with rate A 
= 10 per hour. Calculate the probability that no calls will be received during 
each of two consecutive 15-minute periods. 


Question no. 7 

The customers of a newspaper salesperson arrive according to a Poisson 
process with rate A = 2 per minute. Calculate the probability that at least 
one customer will arrive in the interval (to, to + 2], given that there has been 
exactly one customer in the interval (tọ — 1, tọ + 1], where to > 1. 


Question no. 8 
The stochastic process {X(t),t > 0} is defined by 


A(t) =N(t+1)-N(1) fort >0 


where {N(t),t > 0} is a Poisson process with rate À > 0. Calculate C’x(s,t), 
forO<s<t. 
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Question no. 9 We define 


2,4,... 
1,3,5 


> ee 


_ (rienay=o, 
a + if N(t) = 


where {N(t),t > 0} is a Poisson process with rate A = 1. It can be shown that 
P[Y (t) = 1] = (1+ e7%)/2, for t > 0. Next, let Xn := Y (n) for n =0,1,.... 
Then {Xn,n = 0,1,...} is a Markov chain. Calculate 

(a) its one-step transition probability matrix, 


(b) the limiting probabilities of the chain, if they exist. 


Question no. 10 
The failures of a certain machine occur according to a Poisson process with 
rate À = 1 per week. 


(a) What is the probability that the machine will have at least one failure 
during each of the first two weeks considered? 


(b) Suppose that exactly five failures have occurred during the first four weeks 
considered. Let M be the number of failures during the fourth of these four 
weeks. Calculate E[M | M > 0]. 


Question no. 11 
Let {N(t),¢ > 0} be a Poisson process with rate À > 0. We define 


N,(t)=N(vt), No(t)=N(2t), and Ng(t) = N(t + 2) — N(2) 


and we consider the processes {N;(t),¢ > 0}, for k = 1,2,3. Which of these 
stochastic processes is (or are) also a Poisson process? Justify. 


Question no. 12 

The power failures in a certain region occur according to a Poisson pro- 
cess with rate A; = 1/5 per week. Moreover, the duration X (in hours) of a 
given power failure has an exponential distribution with parameter Ag = 1/2. 
Finally, we assume that the durations of the various power failures are inde- 
pendent random variables. 


(a) What is the probability that the longest failure, among the first three 
power failures observed, lasts more than four hours? 


(b) Suppose that there has been exactly one power failure during the first 
week considered. What is the probability that the failure had still not been 
repaired at the end of the week in question? 


Question no. 13 
A machine is made up of two components that operate independently. 
The lifetime X; (in days) of component i has an exponential distribution with 
parameter À;, for i = 1,2. | 
Suppose that the two components are placed in series and that as soon 
as a component fails, it is replaced by a new one. Let N(t) be the number of 
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replacements in the interval (0,t]. We can show that the stochastic process 
{N(t),t > 0} is a Poisson process. Give its rate À. 


Indication. If Xo is the time between two successive replacements, then we 
can write that Xo = min{ X], Xə}. 


Question no. 14 
We consider the process {X(t),0 < t < 1} defined by 


X(t) = N(#*?) -t*-N(1) for0<t<1 


where {N(t),t > 0} is a Poisson process with rate À > 0. 


(a) Calculate the autocorrelation function, Rx (t1, te), of the stochastic process 
{X(t),0 LS 1} at ti = 1/4 and tg = 1/2. 
(b) Calculate P[X(t) > 0| N(1) = 1], forO<t<1. 


Question no. 15 

Let {N (t), t > 0} be a Poisson process with rate A. Show that, given that 
N(t) =n, N(s) has a binomial distribution with parameters n and s/t, where 
0<s<t, for all à (> 0). 


Question no. 16 
Travelers arrive at a bus station from 6 a.m., according to a Poisson process 
with rate à = 1 per minute. The first bus leaves T minutes after 6 a.m. 


(a) Calculate the mean and the variance of the number of travelers ready to 
board this bus if (i) T has an exponential distribution with mean equal to 15 
minutes and (ii) T is uniformly distributed between 0 and 20 minutes. 


(b) Calculate the average number of passengers on the bus if it leaves at 6:15 
and if its capacity is 20 passengers. 


Question no. 17 

We suppose that every visitor to a museum, independently from the oth- 
ers, moves around the museum for T minutes, where T is a uniform random 
variable between 30 and 90 minutes. Moreover, the visitors arrive according 
to a Poisson process with rate A = 2 per minute. If the museum opens its 
doors at 9 a.m. and closes at 6 p.m., find the mean and the variance of the 
number of visitors in the museum (i) at 10 a.m. and (ii) at time to, where to 
is comprised between 10:30 a.m. and 6 p.m. 


Question no. 18 

Suppose that events occur at random in the plane, in such a way that 
the number of events in a region R is a random variable having a Poisson 
distribution with parameter 1A, where A is a positive constant and A denotes 
the area of the region R. A point is taken at random in the plane. Let D be 
the distance between this point and the nearest event. Calculate the mean of 
the random variable D. 
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Question no. 19 

The various types of traffic accidents that occur in a certain tunnel, over 
a given time period, constitute independent stochastic processes. We suppose 
that accidents stopping the traffic in the northbound (respectively, south- 
bound) direction in the tunnel occur according to a Poisson process with rate 
a (resp., 3). Moreover, accidents causing the complete closure of the tunnel 
occur at rate y, also according to a Poisson process. Let Ty (resp., Tg) be 
the time during which the traffic is not stopped in the northbound (resp., 
southbound) direction, from the moment when the tunnel has just reopened 
after its complete closure. 


(a) Calculate the probability density function of the random variable Ty. 
(b) Show that 


Pity >t,Ts > 7) =e max{ts7} for t 7 > 0 


(c) Check your answer in (a) with the help of the formula in (b). 


Question no. 20 

Let {N*(t),t > 0} be the stochastic process that counts only the even 
events (that is, events nos. 2,4,...) of the Poisson process {N(t),¢ > O} in 
the interval |0,t]. Show that {N*(t),t > 0} is not a Poisson process. 


Question no. 21 

Let {Ni(t),t > 0} and {No(t),t > 0} be two independent Poisson pro- 
cesses, with rates À; and àz, respectively. We define N(t) = Ni (t) — No(t). 
(a) Explain why the stochastic process {N (t), t > 0} is not a Poisson process. 
(b) Give a formula for the probability P[N (t2) — N(ti) = nl, for tg >t; > 0 
and n € {0,+1,+2,...}. 
Question no. 22 | 

Suppose that {N(t),t > 0} is a Poisson process with rate À > 0 and that 
S is a random variable having a uniform distribution on the interval (0, 2]. 
(a) Obtain the moment-generating function of the random variable N(t+ S). 
Indication. If X has a Poisson distribution with parameter a, then Mx (t) = 
exp{a(e’ — 1)}. 
(b) Calculate the mean and the variance of N(t + S). 
Question no. 23 
(a) Is the Poisson process {N(t),t > 0} an ergodic process? Justify. 
(b) Is the stochastic process {X(t),t > 0}, where X(t) := N(t)/t, mean 
ergodic? Justify. 


Question no. 24 
City buses arrive at a certain street corner, between 5 a.m. and 11 p.m., 
according to a Poisson process with rate \ = 4 per hour. Let T} be the 
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waiting time, in minutes, until the first bus (after 5 a.m.), and let M be the 
total number of buses between 5 a.m. and 5:15 a.m. 


(a) Calculate the probability P[T, € I, M = 1], where I := [a,b] is included 
in the interval |0, 15]. 


(b) What is the variance of the waiting time between two consecutive arrivals? 


(c) If a person arrives at this street corner every morning at 9:05 a.m., what 
is the variance of the time during which she must wait for the bus? Justify. 


Question no. 25 

A truck driver is waiting to join the traffic on the service road of a highway. 
The truck driver blocks the service road for four seconds when he joins the 
traffic. Suppose that vehicles arrive according to a Poisson process and that 
six seconds elapse, on average, between two consecutive vehicles. Answer the 
following questions by assuming that the truck driver makes sure that he has 
enough time to perform his maneuver before merging into the traffic. 


(a) What is the probability that the truck driver is able to join the traffic 
immediately on his arrival at the intersection with the service road? 


(b) What is the mathematical expectation of the gap (in seconds) between the 
truck and the nearest vehicle when the truck driver merges into the traffic? 


(c) Calculate the average number of vehicles that the truck driver must let go 
by before being able to merge into the traffic. 


Question no. 26 
Particles are emitted by a radioactive source according to a Poisson process 
with rate A = In5 per hour. 


(a) What is the probability that during at least one of five consecutive hours 
no particles are emitted? 


(b) Knowing that during a given hour two particles were emitted, what is the 
probability that one of them was emitted during the first half-hour and the 
other during the second half-hour of the hour in question? 


(c) In (b), if we know that the first particle was emitted over the first half- 
hour, what is the probability that the second particle was emitted during the 
first half-hour as well? 


Question no. 27 

A system is made up of two components. We suppose that the lifetime (in 
years) of each component has an exponential distribution with parameter A 
= 2 and that the components operate independently. When the system goes 
down, the two components are then immediately replaced by new ones. We 


consider three cases: 


1. the two components are placed in series (so that both components must 
function for the system to work); 
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2. the two components are placed in parallel (so that a single operating com- 
ponent is sufficient for the system to function) and the two components 
operate at the same time; 

3. the two components are placed in parallel, but only one component oper- 
ates at a time and the other is in standby. 


Let N(t), for t > 0, be the number of system failures in the interval (0, t]. 
Answer the following questions for each of the cases above. 


(a) Is {N(t),t > 0} a Poisson process? If it is, give its rate A. If it’s not, justify. 
(b) What is the average time elapsed between two consecutive system failures? 


Question no. 28 

Let {N(t),t > 0} be a Poisson process with rate À = 2 per minute, where 
N(t) denotes the number of customers of a newspaper salesperson in the 
interval [0, t]. 
(a) Let Tk, for k = 1,2,..., be the arrival time of the kth customer. Calculate 
the probability density function of T,, given that To = s. 


(b) Calculate the probability that at least two minutes will elapse until the 
salesperson’s second customer arrives, from a given time instant, given that 
no customers have arrived in the last minute. 


(c) Suppose that the probability that a given customer is a man is equal to 
0.7, independently from one customer to another. Let M be the number of 
consecutive customers who are men before the first woman customer arrives, 
from some fixed time instant tg > 0. Calculate the mathematical expectation 
of M, given that the first customer after tọ was a man. 


Question no. 29 

Suppose that {N(t),t > 0} is a Poisson process with rate A = 2. 
(a) Let Ty be the arrival time of the kth event of the process {N (t), t > 0}, 
for k = 1,2,.... Calculate P[T; + To < T3]. 
(b) Let S be a random variable having a uniform distribution on the interval 
(0, 1] and that is independent of the process {N(t),t > 0}. Calculate E[N?(S)]. 
(c) We define Xn = N(n?), for n = 0,1,2,.... Is the stochastic process 
{Xn n = 0,1,2,...} a Markov chain? Justify by calculating the probability 
PIXn41 =j | Xn = 1,Xn-1 = tn-1;--. , X0 = io]. 
Question no. 30 

We denote by N (t) the number of failures of a machine in the interval [0, t]. 
We suppose that N(0) = 0 and that the time 79 until the first failure has a 
uniform distribution on the interval (0,1]. Similarly, the time Tk- between 
the (k — 1)st and the kth failure has a U(0, 1] distribution, for k = 2,3,.... 
Finally, we assume that To, T1,... are independent random variables. 


(a) Calculate (i) the failure rate of the machine and (ii) PIN(1) = 1]. 
(b) (i) Let rý := -4 In Tk, for k = 0,1,... . We define (see Ex. 5.1.2) 
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n—l 
So =O and S.= > tf forn>1 
k=0 


Finally, we set 
N(= makin > 0:S,< 1} 


Show that the stochastic process {N*(t),t > 0} is a Poisson process, with 
rate A = 2, by directly calculating the probability density function of Tý, for 
all k, with the help of Proposition 1.2.2. 

(ii) Use the central limit theorem to calculate (approximately) the proba- 
bility P{S100 > 40}. 


Question no. 31 
In the preceding question, suppose that the random variables 7, all have 
the density function 


f(s) = se~* /? fors>0 


(a) (i) Now what is the failure rate of the machine and (ii) what is the value 
of the probability P[N(1) = 1]? 
Indication. We have 


1 
J tet ttdt ~ 0.5923 
Q 


(b) (i) Let T := TŽ, for k = 0,1,... . We define the random variables S, and 
the stochastic process {N*(t),t > 0} as in the preceding question. Show that 
the process {N*(t),¢ > 0} is then a Poisson process with rate A = 1/2. 

(ii) Calculate approximately the probability P[So5 < 40] with the help of 
the central limit theorem. 


Question no. 32 

A woman working in telemarketing makes telephone calls to private homes 
according to a Poisson process with rate A = 100 per (working) day. We 
estimate that the probability that she succeeds in selling her product, on a 
given call, is equal to 5%, independently from one call to another. Let N(t) be 
the number of telephone calls made in the interval [0, t], where t is in (working) 
days, and let X be the number of sales made during one day. 


(a) Suppose that the woman starts her working day at 9 a.m. and stops 
working at 7 p.m. Let Tọ be the number of minutes between 9 a.m. and the 
moment of her first call of the day, and let Sp be the duration (in minutes) of 
this call. We suppose that Sọ ~ Exp(1) and that Tọ and Sp are independent 
random variables. What is the probability that the woman has made and 
finished her first call at no later than 9:06 a.m. on an arbitrary working day? 


(b) Calculate V[X | N(1) = 100]. 
(c) Calculate (approximately) P[N(1) = 100 | X = 5]. 
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ee a ie 
Indication. Stirling’s* formula: n! ~ /2an"tze—™. 


(d) What is the probability that the woman will make no sales at all on exactly 
one day in the course of a week consisting of five working days? 


Question no. 33 

The breakdowns of a certain device occur according to a Poisson process 
with rate À = 2 per weekday, and according to a Poisson process (independent. 
of the first process) with rate À = 1.5 per weekend day. Suppose that exactly 
four breakdowns have occurred (in all) over two consecutive days. What is 
the probability that both days were weekdays? 


Question no. 34 

Let {N (t), t > 0} be a Poisson process with rate À = 2 per minute. What 
is the probability that the time elapsed between at least two of the first three 
events of the process is smaller than or equal to one minute? 


Question no. 35 

Let N (t) be the number of telephone calls to an emergency number in the 
interval [0,t]. We suppose that {N (t), t > 0} is a Poisson process with rate A 
= 50 per hour. 
(a) What is, according to the model, the probability that there are more calls 
from 8 a.m. to 9 a.m. than from 9 a.m. to 10 a.m.? 
Indication. If X ~ Poi(50) and Y ~ Poi(50) are independent random variables, 
then we have that P[X = Y] ~ 0.0399. 
(b) Suppose that 20% of the calls are redirected to another service. Knowing 
that there were 60 (independent) calls during a given hour and that the first 
of these calls was redirected elsewhere, what is the mathematical expectation 
of the number of calls that were redirected over the hour in question? 


Question no. 36 
From a Poisson process with rate A, {N(t),t > 0}, we define the stochastic 
process {M (t), t > 0} as follows: 


M(t) = N(t)—t fort >0 


Calculate P[S1 < 2], where Sı := min{t > 0: M(t) > 1}. 


Question no. 37 


Let 
N(t +6) — N(t) 


6 
where {N(t),t > 0} is a Poisson process with rate A and 6 > 0 is a constant. 
(a) Calculate ELX (t)]. 


X(t) := for t > 0 


* See p. 93. 
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(b) Calculate Cov[X (s), X(t)], for s,t > 0. 
(c) Is the process {X(t),t > 0} wide-sense stationary? Justify. 


Question no. 38 

A machine is composed of three identical components placed in standby 
redundancy, so that the components operate (independently from each other) 
by turns. The lifetime (in weeks) of a component is an exponential random 
variable with parameter à = 1/5. There are no spare components in stock. 
What is the probability that the machine will break down at some time during 
the next nine weeks, from the initial time, and remain down for at least a week, 
if we suppose that no spare components are expected to arrive in these next 
nine weeks? 


Question no. 39 
Let {X(t), t > 0} be the stochastic process defined by 


X(t) =tN(t) —[e\N((d]) for t > 0 


where {N(t),t > 0} is a Poisson process with rate A (> 0) and ft) denotes the 
integer part of t. 

(a) Is the process {X (t), t > 0} a continuous-time Markov chain? Justify. 

(b) Calculate the probability p; ;(t,t + s) := P[X(t+s) = j | X(t) = i], for 
i,j € {0,1,...} and s,t > 0. 

Question no. 40 


We consider the stochastic process {X (t), t > 0} defined from a Poisson 
process with rate A, {N (t), t > 0}, as follows (see Ex. 5.1.4): 


X({t)= N(t)-—tN(1) fr0<t<1 


(a) Let M be the number of visits of the stochastic process {X(t),t > 0} to 
any state x; > 0, from any state x; < 0, in the interval 0 < t < 1. Calculate 
P|M =n|N(1)=nl. 
(b) We define T = min{t > 0: X(t) > 0}. Calculate the conditional probabil- 
ity density function fr(t | N(1) = 2). 
Question no. 41 

Consider the stochastic process {Y(t),t > 0} defined in Question no. 9, 
where {N(t),t > 0} is now a Poisson process with an arbitrary rate A (> 0). 
It can be shown that we then have that P[Y (t) = 1] = (1+e7**)/2, for t > 0. 
Calculate 
(a) PIN(2) - N(1) >1|N(Q) =4, 
(b) PIN(t) =0| Y(t) = J], 
(c) PIY (s) =1| N(¢) = 1], whereO< s <t. 


300 5 Poisson Processes 


Question no. 42 
We set 
X(t)= N(t+c)—N(e) fort >0 


where {N(t),t > 0} is a Poisson process with rate à and c is a positive 
constant. Is the stochastic process {X(t),t > 0} a Poisson process? Justify. 


Question no. 43 
Let {N(t),t > 0} be a Poisson process with rate à > 0 and let {X(¢), 
t > 0} be the stochastic process defined as follows: 


N(t +62) — N(t) 


for t > 0 
; or t > 


X(t) = lim 


where ô is a positive constant. 
(a) Calculate Cov[X (t1), X(t2)}, for t1,t2 > 0. 
(b) Is the process {X(t),¢t > 0} wide-sense stationary? Justify. 


Question no. 44 

Suppose that {N(t),t > 0} is a Poisson process with rate A. Let Xn := 
N?(n), for n = 0,1,... . Is the stochastic process {Xn n =0,1,...} a discrete- 
time Markov chain? If it is, give its one-step transition probability matrix. If 
it’s not, justify. 


Question no. 45 
Let Tk be the arrival time of the kth event of the Poisson process {N (t), 
t > 0}, with rate À, for k = 1,2,... . Calculate the covariance Cov[T, Tol. 


Question no. 46 

We consider a Poisson process, {N (t), t > 0}, with rate A = 1. Let T), 
To, ... be the arrival times of the events and let S4 := Ti, So := To — Ti, 
S3 := T3 — To, etc. 
(a) Calculate PH{ S1 < S2 < S3} U {S3 < Sp < S1}. 
(b) Let X := ga, Calculate (i) fx(x) and (ii) PIX < S2]. 
(c) Calculate E[N? (1) | Te = 5]. 


Question no. 47 

Let {Xn, n = 0,1,... } be a (discrete-time) Markov chain whose state space 
is the set Z of all integers. Suppose that the process spends an exponential 
time 7 (in seconds) with parameter A = 1 in each state before making a 
transition and that the next state visited is independent of r. Let N(t), for 
t > 0, be the number of transitions made in the interval (0, t]. 


(a) What is the probability that the third transition took place before the 
fifth second, given that five transitions occurred during the first 10 seconds? 


(b) Calculate E[|N (5) — 1]]. 
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Sections 5.2 to 5.5 


Question no. 48 
The failures of a certain device occur according to a nonhomogeneous 
Poisson process whose intensity function A(t) is given by 


0.2if0<¢< 10 
At) = or ift > 10 


where t is the age (in years) of the device. 


(a) Calculate the probability that a five-year-old device will have exactly two 
failures over the next 10 years. 


(b) Knowing that the device had exactly one failure in the course of the first 
ö years of the 10 years considered in (a), what is the probability that this 
failure took place during its sixth year of use? 


Question no. 49 

Suppose that traffic accidents occur in a certain region according to a 
Poisson process with rate À = 2 per day. Suppose also that the number M of 
persons involved in a given accident has a geometric distribution with param- 
eter p = 1/2. That is, 


P Mem S (727 form 21,2735 


(a) Calculate the mean and the variance of the number of persons involved in 
an accident over an arbitrary week. 


(b) Let T be the random variable denoting the time between the first person 
and the second person involved in an accident, from the initial time. Calculate 
the distribution function of T. 


Question no. 50 

Suppose that the monthly sales of a dealer of a certain luxury car constitute 
a conditional Poisson process such that A is a discrete random variable taking 
the values 2, 3, or 4, with probabilities 1/4, 1/2, and 1/4, respectively. 
(a) If the dealer has three cars of this type in stock, what is the probability 
that the three cars will be sold in less than a month? 


(b) Suppose that A = 3. Calculate VIM | M < 1], where M is the number of 
cars sold in one month. 


Question no. 51 

We suppose that the traffic at a point along a certain road can be described 
by a Poisson process with parameter À = 2 per minute and that 60% of the 
vehicles are cars, 30% are trucks, and 10% are buses. We also suppose that the 
number K of persons in a single vehicle is a random variable whose function 
PK is 


302 5 Poisson Processes 


1/2ifk=1 

1/4ifk=2 0.9ifk=1 
Px(k) = athe ne To th a2 

1/8 if k =4 


in the case of cars and trucks, respectively, and px(k) = 1/50, for k = 
1,... ,50, in the case of buses. 


(a) Calculate the variance of the number of persons who pass by this point in 
the course of a five-minute period. 


Indication. We have n 
5 L2 n(n + 1)(2n +1) 
k=1 6 


(b) Given that five cars passed by the point in question over a five-minute 
period, what is the variance of the total number of vehicles that passed by 
that point during these five minutes? 


Indication. We assume that the number of cars is independent of the number 
of trucks and buses. 


(c) Calculate, assuming as in (b) the independence of the vehicles, the prob- 
ability that two cars will pass by this point before two vehicles that are not 
cars pass by there. 


(d) Suppose that actually 


t/5 if0<t<10 
\= X(t) = 2 if10<t<50 
(60 — t)/5 if 50 < t < 60 


Calculate the probability PIN (60) = 100 | N(30) = 60], where N (t) is the 
total number of vehicles in the interval ÍO, t]. 


Question no. 52 

Let N (t) be the number of accidents at a specific intersection in the interval 
10, t]. We suppose that {N(t),t > 0} is a Poisson process with rate A; = 1 
per week. Moreover, the number Y; of persons injured in the kth accident has 
(approximately) a Poisson distribution with parameter Ag = 1/2, for all k. 
Finally, the random variables Y1, Y2,... are independent among themselves 
and are also independent of the stochastic process {N (t), t > 0}. 


(a) Calculate the probability that the total number of persons injured in the 
interval [0,t¢] is greater than or equal to 2, given that N(t) = 2. 

(b) Calculate V[N(t) Yi]. 

(c) Let S be the time instant when the kth person was injured, for k = 
1,2,.... We set T = Sq — Sı. Calculate PÍT > 0]. 


Question no. 53 
Let {N(t),t > 0} be a Poisson process with rate L. 
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(a) Suppose that L is a random variable having an exponential distribution 
with parameter 1 (so that {N(t),t > 0} is actually a conditional Poisson 
process). Let Tı be the arrival time of the first event of the stochastic process 
{N(t),t > O}. 

(i) Calculate the probability density function of T}. 

(ii) Does the random variable T, have the memoryless property? Justify. 


(b) Let L = 1/2. Calculate E[N?(1) | NG) < 2]. 
(c) Let L = 1. Calculate P{N(3) — N(1) > 0| N(2) = 1]. 


Question no. 54 
Telephone calls to an emergency number arrive according to a nonhomo- 
geneous Poisson process whose intensity function is given by 


ney = [2E0<t<6 (by night) 
7 | 4if 6 <t < 24 (by day) 


where ¢ is in hours, and A(t) = A(t — 24), for t > 24. Furthermore, the 
duration (in minutes) of a call received at night has a uniform distribution on 
the interval (0, 2], whereas the duration of a call received during the day has 
a uniform distribution on the interval (0, 3]. Finally, the durations of calls are 


independent random variables. 

(a) Calculate the probability that an arbitrary telephone call received at night 
will be longer than a given call received during the day. 

(b) Calculate the variance of the number of calls received in the course of a 
given week. 


(c) Let D be the total duration of the calls received during a given day. 
Calculate the variance of D. 


Question no. 55 

Let {N(t),¢ > 0} be a Poisson process with rate À = 2. Suppose that all 
the events that occur in the intervals (2k,2k +1], where k € {0,1,2,...}, are 
counted, whereas the probability of counting an event occurring in an interval 
of the form (2k + 1, 2k + 2] is equal to 1/2. Let Ny(t) be the number of events 
counted in the interval f0, t]. 


(a) Calculate P[N,(2.5) > 2]. 
(b) Calculate V[N (2) — 2N1(1)]. 


(c) Let Sı be the arrival time of the first counted event. Calculate 


P[S, < s | Nı(2)=1] Vse (0,2] 


Question no. 56 

In the preceding question, suppose that À = 1 and that the probability of 
counting an event occurring in an interval of the form (2k +1,2k + 2] is equal 
to (1/2)*, for k € {0,1,2,...}. 
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(a) Calculate P[N1(5) > 5 | Ny (2) = 2]. 

(b) Calculate Cov[Ni(5),Ni(2)]. | | 

(c) Let M be the total number of counted events in the intervals of the form 
(2k +1,2k + 2}. Calculate E[M]. 

(d) Let S be the arrival time of the first event in an interval of the form 
(2k, 2k + 1]. Calculate fs(s | N(3) — N(2) + N(1) = 1). 


Question no. 57 
Let {Ni(t),¢ > 0} be a (homogeneous) Poisson process with rate A = 1, 
and let {No(t),t > 0} be a nonhomogeneous Poisson process whose intensity 


function is ‘ aa 
1ifo<t< 
Me) = {oes 


We suppose that the two stochastic processes are independent. Moreover, let 
{Y (t),t > 0} be a compound Poisson process defined by 


Ni(t) 
Y(t)= Ý X; (and Y(0) = 0 if M(t) = 0) 
i=1 


where X; has a Poisson distribution with parameter 1, for all i. Calculate 
(a) PITi 2 < To,2,.Ni(1) = No(1) = 0], where Tm,n denotes the arrival time of 
the nth event of the process {N,,(t),¢ > 0}, for m = 1,2 and n = 1,2,..., 
(b) V[No(1)(N2(2) — No(1))), 

(c) (i) P[Y(1) < Ni(1) < JJ; (ii) V[Y (5) | N1(5) < 2]. 

Question no. 58 


Suppose that the intensity function A(t) of the nonhomogeneous Poisson 
process {N(t),t > 0} is given by 





A(t): = : TA for t > 0 


t2 4 
where A > 0 and t is in minutes. 

(a) Let T; be the arrival time of the first event of the stochastic process 
{N(t),t > 0}. Calculate P[T, < s | N(1) = 1], for s € (0, 1]. 

(b) Suppose that A = 2 and that N(5) > 2. Calculate the probability that, at 
time tp = 10, at least five minutes have elapsed since the penultimate event 
occurred. 


Question no. 59 
In the preceding question, suppose that 


1 
= —— fort>0 
A(t) I+ or t > 


where t is in minutes. 


5.7 Exercises 305 


(a) Calculate fr,(s | N(1) = 1), for s € (0,1]. 

(b) Suppose that N(5) > 3. Calculate the probability that, at time tọ = 10, 
at least five minutes have elapsed since the antepenultimate (that is, the one 
before the penultimate) event occurred. 


Question no. 60 3 
The stochastic process {Y (t), t > 0} is a compound Poisson process defined 
by 
N(t) 
Y=) Xe (and Y(t) =0 if N(t) = 0) 
k=1 


where X;, has a geometric distribution with parameter 1/4, for k = 1,2,..., 
and {N(t),t > 0} is a Poisson process with rate À = 3. 


(a) Calculate ETY (t) | Y(t) > 0], for t > 0. 


(b) Calculate approximately P[Y (10) > 100} with the help of the central limit 
theorem. 


Question no. 61 

Suppose that, in the preceding question, X; is a discrete random variable 
such that P[X,; = 1] = P[X, = 2] = 1/2, fork =1,2,..., and that {N(¢),t > 
0} is a Poisson process with rate A = 2. 
(a) Calculate E[Y?(t) | Y(t) > 0], for t > 0. 


(b) Use the central limit theorem to calculate (approximately) the probability 
P{Y (20) < 50]. 


Question no. 62 

The (independent) visitors of a certain Web site may be divided into two 
groups: those who arrived on this site voluntarily (type I) and those who 
arrived there by chance or by error (type I). Let N (t) be the total number 
of visitors in the interval {0,t]. We suppose that {N(t),¢ > 0} is a Poisson 
process with rate À = 10 per hour, and that 80% of the visitors are of type I 
(and 20% of type II). 


(a) Calculate the mean and the variance of the number of visitors of type I, 
from a given time instant, before a second type II visitor accesses this site. 


(b) Calculate the variance of the total time spent on this site by the visitors 
arrived in the interval [0, 1] if the time (in minutes) Xz (respectively, X77) that 
a type I (resp., type II) visitor spends on the site in question is an exponential 
random variable with parameter 1/5 (resp., 2). Moreover, we assume that X7 
and X77; are independent random variables. 


(c) Suppose that, actually, {N(t),t > 0} is a nonhomogeneous Poisson process 
whose intensity function is 


5 if0<t<7 
Mt ={ soe 7 et eo 
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and A(t+24n) = X(t), for n = 1,2,... . Given that exactly one visitor accessed 
this site between 6 a.m. and 8 a.m., what is the distribution function of the 
random variable S denoting the arrival time of this visitor? 


Question no. 63 

At night, vehicles circulate on a certain highway with separate roadways 
according to a Poisson process with parameter À = 2 per minute (in each di- 
rection). Due to an accident, traffic must be stopped in one direction. Suppose 
that 60% of the vehicles are cars, 30% are trucks, and 10% are semitrailers. 
Suppose also that the length of a car is equal to 5 m, that of a truck is equal 
to 10 m, and that of a semitrailer is equal to 20 m. 


(a) From what moment is there a 10% probability that the length of the queue 
of stopped vehicles is greater than or equal to one kilometer? 


(b) Give an exact formula for the distribution of the length of the queue of 
stopped vehicles after t minutes. 


Indication. Neglect the distance between the stopped vehicles. 


Question no. 64 

During the rainy season, we estimate that showers, which significantly 
increase the flow of a certain river, occur according to a Poisson process with 
rate À = 4 per day. Every shower, independently from the others, increases 
the river flow during T days, where T is a random variable having a uniform 
distribution on the interval [3,9]. 


(a) Calculate the mean and the variance of the number of showers that sig- 
nificantly increase the flow of the river 

(i) six days after the beginning of the rainy season, 

(ii) to days after the beginning of the rainy season, where to > 9. 


(b) Suppose that every (significant) shower increases the river flow by a quan- 
tity X (in m?°/s) having an exponential distribution with parameter 1/10, 
independently from the other showers and from the number of significant 
showers. Suppose also that there is a risk of flooding when the increase in the 
river flow reaches the critical threshold of 310 m?/s. Calculate approximately 
the probability of flooding 10 days after the beginning of the rainy season. 


Question no. 65 

Independent visitors to a certain Web site (having infinite capacity) arrive 
according to a Poisson process with rate À = 30 per minute. The time that a 
given visitor spends on the site in question is an exponential random variable 
with mean equal to five minutes. Let Y(t) be the number of visitors at time 
t > 0. The stochastic process {Y(t),t > 0} is a filtered Poisson process. 


(a) What is the appropriate response function? 
(b) Calculate E[Y(t)] and V[Y(t)]. 
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Question no. 66 
Let {Ni(t),¢ > 0} and {No(t),t > 0} be independent Poisson processes, 
with rates A; and Az, respectively. 


(a) We denote by 7) the arrival time of the first event of the stochastic 
process {N2(t),t > 0}. Calculate E[Ni(T21) | Ni(T2,1) < 2]. 


(b) We define M(t) = Ni (t) + No(t) and we set 


1 if the kth event of the process {M (t), t > 0} 
is an event of {N1 (t),t > 0} 
N= 
0 if the kth event of the process {M(t),t > 0} 
is an event of {No(t),t > 0} 


Calculate V|Y (t)], where 


M(t) 
Y(t):= S> Xe (and Y(t) = 0 if M(t) =0) 
k=1 


(c) Suppose that {N2(t),t > 0} is rather a nonhomogeneous Poisson process 
whose intensity function is 


1 ifO0<t<1 
alt) =f yyy ift>1 


Calculate P|T2 < 2}, where Tz, is defined in (a). 


Section 5.6 


Question no. 67 

A system is composed of two components that operate at alternate times. 
When component i, for i = 1,2, starts to operate, it is active during X; 
days, where X; is an exponential random variable with parameter A; and is 
independent of what happened previously. The state of the components is 
checked only at the beginning of each day. If we notice that component í 
is down, then we set the other component going, and component 7 will be 
repaired (in less than one day). 
(a) Let N; be the number of consecutive days during which component i 
is responsible for the functioning of the system, for i = 1,2. What is the 
probability distribution of N;? 
(b) Suppose that the two components are identical. That is, Ay = Àz := À. At 
what rate do the components relieve each other (over a long period)? 
(c) If Ay = 1/10 and Az = 1/12, what proportion of time, when we consider a 
long period, is component 1 responsible for the functioning of the system? 
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Question no. 68 

A woman makes long-distance calls with the help of her cell phone accord- 
ing to a Poisson process with rate À. Suppose that for each long-distance call 
billed, the next call is free and that we fix the origin at the moment when a 
call has just been billed. Let N(t) be the number of calls billed in the interval 
(0, ¢]. 
(a) Find the probability density function of the random variables 79,7),..., 
where To is the time until the first billed call, and 7; is the time between the 
(i — 1)st and the ith billed call, for ¿i > 1. 


(b) Calculate the probability PIN (t) = n], for n =0,1,.... 
(c) What is the average time elapsed at time t since a call has been billed? 


Question no. 69 

A machine is made up of two independent components placed in series. 
The lifetime of each component is uniformly distributed over the interval [0, 1]. 
As soon as the machine breaks down, the component that caused the failure is 
replaced by a new one. Let N(t) be the number of replacements in the interval 
(0, t]. 
(a) Is the stochastic process {N (t), t > 0} a continuous-time Markov chain? 
Justify. 


(b) Is {N(t),¢ > 0} a renewal process? Justify. 


(c) Let Sı be the time of the first replacement. Calculate the probability 
density function of Sj. 


Question no. 70 (See Question no. 98, p. 171) 

The lifetime of a certain machine is a random variable having an expo- 
nential distribution with parameter A. When the machine breaks down, there 
is a probability equal to p (respectively, 1 — p) that the failure is of type I 
(resp., II). In the case of a type I failure, the machine is out of use for an 
exponential time, with mean equal to 1/j time unit(s). To repair a type H 
failure, two independent operations must be performed. Each operation takes 
an exponential time with mean equal to 1/4. 


(a) Use the results on regenerative processes to calculate the probability that 
the machine will be in working state at a (large enough) given time instant. 


(b) What is the average age of the machine at time t? That is, what is the 
average time elapsed at time t since the most recent failure has been repaired? 
Assume that A = u. 


Question no. 71 
We consider a discrete-time Markov chain whose state space is the set 
{0,1,2} and whose one-step transition probability matrix is 


0 1/21/2 
P=|1/2 0 1/2 
1 0 0 
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We say that a renewal took place when the initial state 0 is revisited. 


(a) What is the average number of transitions needed for a renewal to take 
place? 
(b) Let N(t), for t > 0, be the number of renewals in the interval [0, t], where 
t is in seconds. If we suppose that every transition of the Markov chain takes 
one second, calculate 

(i) the distribution of the random variable N(6.5), 

(ii) the probability P[N(90) < 40] (approximately). 


Question no. 72 
We consider a birth and death process, {X(t),t > 0}, whose state space 
is the set {0,1} and for which 


Ag =A and pu =p 


Moreover, we suppose that X(0) = 0. We say that a renewal occurred when 
the initial state 0 is revisited. Let N(t), for t > 0, be the number of renewals 
in the interval [0, t]. 

(a) Let T,-1 be the time between the (n — 1)st and the nth renewal, for 
n > 1 (To being the time until the first renewal). Find the probability density 
function of T,_1 if A = pu. 

(b) Calculate approximately the probability P[N(50) < 15] if A = 1/3 and 
=l. 

(c) Calculate the average time elapsed at a fixed time instant tp since a renewal 
occurred if A= 2 and p= 1. 


Question no. 73 
Let {X(t),t > 0} be a birth and death process for which 


An =A Vn>0 and fy =p, Un=0 Vn22 


We suppose that X(0) = 0 and we say that a renewal took place when the 
process revisits the initial state 0. We denote by N(t), for t > 0, the number 
of renewals in the interval [0, t]. 
(a) Let Tı be the time elapsed until the first renewal, and let M(t) be the 
number of deaths in the interval [0, t]. Calculate 

(ii) Pilimi M(t) = k], for k € {0,1,2,...}. 
(b) Is the stochastic process {X(t),t > 0} a regenerative process? Justify. 
(c) Suppose now that Aj = A and A,, = 0 if n > 1. Calculate the proportion 
of time that the process {X(t),¢ > 0} spends in state 0, over a long period. 


Question no. 74 

Let {N(t),¢ > 0} be a renewal process for which the time r between the 
successive events is a continuous random variable whose probability density 
function is given by 
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fr(s) =se for s>0 
That is, 7 has a gamma distribution with parameters a = 2 and A = 1. 


(a) Show that the renewal function, my(t), is given by 


L 
mn (t) = rh +2t-—1) fort>0 


Indication. We have 


oo 2n-1 


T i £ gT 
D eT a =e) 


n=1 


(b) Calculate approximately the probability P[Ty(100)41 < 101.5]. 


(c) Let X(t) := (-1)%™, for t > 0. The stochastic process {X (t), t > 0} isa 
regenerative process. Calculate the limiting probability that X(t) = 1. 


Question no. 75 

Consider the renewal process {N(t),t > 0} for which the time 7 between 
the consecutive events is a continuous random variable having the following 
probability density function: 


f(s) =2s forO<s<l 


(a) Calculate the renewal function, my (t), forO <¢ <1. 


Indication. The general solution of the second-order ordinary differential equa- 
tion y” (x) = ky(x) is 


1/2 _ 1/2 
y(z) = ciet = + ege" * 


where k Æ 0, and cı and cz are constants. 


(b) According to Markov’s inequality, what is the maximum value of the 
probability P[Ty(1/2)41 2 1]? 


(c) Suppose that we receive a reward of $1 per time unit when the age, A(t), of 
the renewal process is greater than or equal to 1/2, and of $0 when A(t) < 1/2. 
Calculate the average reward per time unit over a long period. 


Question no. 76 

The time between the successive renewals, for a certain renewal process 
{N(t),t > 0}, is a continuous random variable whose probability density 
function is the following: | 


_ f1/2if0<s<1/2 
o ae 
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(a) We can show that my(t) = e/* — 1, for t € (0,1/2). Use this result to 
calculate my (t), for t € [1/2,1). 


Indication. The general solution of the ordinary differential equation 


y (x) + P(x)y(x) = Q(x) 


y(x) =e aie aaa Peer ae constant 


(b) Use the central limit theorem to calculate approximately P[To5 > 15], 
where JT»; is the time of the 25th renewal of the process {N(t),t > 0}. 


(c) Let {X(t),t > 0} be a regenerative process whose state space is the set 
{1,2}. We suppose that the time that the process spends in state 1 is a random 
variable Y; such that 


_ fi1/rifo<y<1/2 
Ivy) = ee if1/2<y<1 


while the probability density function of the time Y2 that the process spends 


in state 2 is j 
_J32iosy< i 
fry) = ay if1/2<y<1 


Calculate lim: P[X(t) = I]. 


Question no. 77 

We consider a system composed of two subsystems placed in series. The 
first subsystem comprises a single component (component no. 1), whereas the 
second subsystem comprises two components placed in parallel. Let Sẹ be the 
lifetime of component no. k, for k = 1,2,3. We assume that the continuous 
random variables Są are independent. 


(a) Let N(t) be the number of system failures in the interval [0, t]. In what 
case(s) will the stochastic process {N(t),t > 0} be a renewal process if the 
random variables S do not all have an exponential distribution? Justify. 


(b) In the particular case when the variable Sẹ has an exponential distribution 
with parameter 41 = 1, for all k, the process {N(t),t > 0} is a renewal 
process. Calculate the probability density function of the time r between two 
consecutive renewals. 


Question no. 78 
Suppose that {N(t),t > 0} is a renewal process for which the time T 
between two consecutive renewals is a continuous random variable such that 


ae — if0<s<l 
0 elsewhere 
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(a) Calculate the renewal function, my(t), forO<t <1. 
Remark. See the indication for Question no. 76. 


(b) If we receive a reward of $1 at the moment of the nth renewal if the 
duration of the cycle has been greater than 1/2 (and $0 otherwise), what is 
the average reward per time unit over a long period? 


Question no. 79 

We consider a system made up of two subsystems placed in parallel. The 
first subsystem is composed of two components (components nos. 1 and 2) 
placed in parallel, while the second subsystem comprises a single component 
(component no. 3). Let 5; denote the lifetime of component no. k, for k = 
1,2,3. The continuous random variables Sọ are assumed to be independent. 


(a) Suppose that components nos. 1 and 2 operate at the same time, from the 
initial time, whereas component no. 3 is in standby and starts operating when 
the first subsystem fails. When the system breaks down, the three components 
are replaced by new ones. Let N(t), for t > 0, be the number of system failures 
in the interval [0,¢]. Then {N(t),t > 0} is a renewal process. Let r be the 
time between two consecutive renewals. Calculate the mean and the variance 
of 7 if Sk ~ U(0,1), for k = 1, 2,3. 

(b) Suppose that we consider only the first subsystem and that the two com- 
ponents are actually placed in series. When this subsystem fails, the two 
components are replaced by new ones. As in (a), the process {N(t),t > 0} 
is a renewal process. Calculate the renewal function my(t), for 0 < ¢ < 1, if 
Sk ~ U(0,1), for k = 1,2. 


Indication. The general solution of the differential equation 
y” (x) — 2y"(x) + 2y(z) +2 = 0 
is 
y(x) = —1 + ce” cos z + coe” sing 
where cı and cz are constants. 


(c) Suppose that in (b) we replace only the failed component when the sub- 
system breaks down and that Sẹ ~ Exp(2), for k = 1,2. 

(i) Calculate the mean of Tyy(4)41 — t. 

(ii) Deduce from it the value of my (t), for t > 0. 


Question no. 80 

Is a nonhomogeneous Poisson process with intensity function A(t) = t, 
for all t > 0, a renewal process? If it is, give the distribution of the random 
variables 7%. If it’s not, justify. 


Question no. 81 

Use the renewal equation to find the distribution of the random variables 
Tk, taking their values in the interval [0, 7/2], of a renewal process for which 
my (t) = t?/2, for 0 < t < 7/2. 
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Question no. 82 

Let {X(t),t > 0} be a birth and death process with state space {0, 1,2} 
and having the following birth and death rates: Ay = A, = A > 0, uı unknown, 
and u2 = p > 0. For what value(s) of u1 is the process regenerative? Justify. 


Question no. 83 

Suppose that {N(t),t > 0} is a Poisson process with rate \ > 0. For a 
fixed time instant t > 0, we consider the random variables A(t) and D(t) (see 
p. 280). 
(a) Calculate the distribution of both A(t) and D(t). 
(b) What is the distribution of A(t) + D(t)? 
(c) Suppose now that we consider the Poisson process over the entire real line, 
that is, {N(t),t € R}. What then is the distribution of A(t) + D(t)? 


(d) In (c), we can interpret the sum A(t) + D(t) as being the length of the 
interval, between two events, which contains the fixed time instant t. Explain 
why the distribution of this sum is not an exponential distribution with pa- 
rameter À. 


Question no. 84 

Let {B(t),t > 0} be a standard Brownian motion, that is, a Wiener process 
with drift coefficient u = 0 and diffusion coefficient o? = 1. Suppose that 
when B(t) = a (> 0) or b = —a, the process spends an exponential time with 
parameter À = 2 in this state (a or b = —a). Then, it starts again from state 
0. 


(a) What fraction of time does the process spend in state a or b = —a, over a 
long period? 


Indication. Let mı(x) be the average time that the process, starting from 
x € (b,a), takes to reach a or b. We can show that the function m(x) satisfies 
the ordinary differential equation 


sm (2) = —1 (with mı (b) = mı (a) = 0) 


(b) Answer the question in (a) if b = —oo rather than —a. 


Question no. 85 


Suppose that the time between two consecutive events for the renewal process 
{N (t), t > 0} is equal to 1 with probability 1/2 and equal to 2 with probability 
1/2. 

(a) Give a general formula for the probability P[N(2n) = k], where n and k 
are positive integers. 

(b) Calculate my (3). 


(c) Let I(t) := 1 if N(t) = [t] and I(t) = 0 otherwise, where [ | denotes the 
integer part. Calculate the variance of I(t). 


6 


Queueing Theory 


6.1 Introduction 


In this chapter, we will consider continuous-time and discrete-state stochastic 
processes, {X(t),t > 0}, where X(t) represents the number of persons in 
a queueing system at time t. We suppose that the customers who arrive in 
the system come to receive some service or to perform a certain task (for 
example, to withdraw money from an automated teller machine). There can 
be one or many servers or service stations. The process {X(t),t > 0} is a 
model for a queue or a queueing phenomenon. If we want to be precise, the 
queue should designate the customers who are waiting to be served, that is, 
who are queueing, while the queueing system includes all the customers in 
the system. Since queue is the standard expression for this type of process, 
we will use these two expressions interchangeably. Moreover, it is clear that 
the queueing models do not apply only to the case when we are interested in 
the number of persons who are waiting in line. The customers in the system 
may be, for example, airplanes that are landing or are waiting for the landing 
authorization, or machines that have been sent to a repair shop, etc. 

Kendall! proposed, in a research paper published in 1953, a notation to 
classify the various queueing models. The most general notation is of the form 
A/S/s/c/p/D, where 


A denotes the distribution of the time between two successive arrivals, 
S denotes the distribution of the service time of customers, 

s is the number of servers in the system, 

c is the capacity of the system, 

p is the size of the population from which the customers come, 

D designates the service policy, called the discipline, of the queue. 


‘ David George Kendall, retired professor of the University of Cambridge, in Eng- 
land. 
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We suppose that the times 7, between the arrivals of successive customers 
are independent and identically distributed random variables. Similarly, the 
service times S, of the customers are random variables assumed to be i.i.d. 
and independent of the 7,’s. Actually, we could consider the case when these 
variables, particularly the S,,’s, are not independent among themselves. 

The most commonly used distributions for the random variables 7, and 
Sn, and the corresponding notations for A or S, are the following: 


M exponential with parameter A or p; 
Ep Erlang (or gamma) with parameters k and À or p; 
D degenerate or deterministic (if Tn or Sn is a constant); 


G general (this case includes, in particular, the uniform distribution). 


Remarks. i) We write M when Tn (respectively, Sn) has an exponential distri- 
bution, because the arrivals of customers in the system (resp., the departures 
from the system in equilibrium) then constitute a Poisson process, which is a 
Markovian process. 


ii) We can use the notation GI for general independent, rather than G, to be 
more precise. 


The number s of servers is a positive integer, or sometimes infinity. (For 
example, if the customers are persons arriving in a park and staying there 
some time before leaving for home or elsewhere, in which case, the customers 
do not have to wait to be served.) 

By default, the capacity of the system is infinite. Similarly, the size of the 
population from which the customers come is assumed to be infinite. If e (or 
p) is not equal to infinity, its value must be specified. On the other hand, 
when c = p = &, we may omit these quantities in the notation. 

Finally, the queue discipline is, by default, that of first-come, first-served, 
which we denote by FCFS or by FIFO, for first-in, first-out. We may also 
omit this default discipline in the notation. In all other cases, the service 
policy used must be indicated. We can have LIFO, that is, last-in, first-out. 
The customers may also be served at random (RANDOM). Sometimes one or 
more special customers are receiving priority service, etc. 

In this book, except in the penultimate subsection of the current aha 
ter and some exercises, we will limit ourselves to the case when the random 
variables 7, and S, have exponential distributions, with parameters A and p, 
respectively. That is, we will only study models of the form M/M. Moreover, 
in the text, the service policy will be the default one (FIFO). However, in the 
exercises, we will often modify this service policy. 

For all the queueing systems that we will consider, we may assume that 
the limit 


= lim P[X(t) = n] (6.1) 
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exists, for all n > 0. Thus, 7, designates the limiting probability that there 
are exactly n customers in the system. Moreover, mn is also the proportion of 
time when the number of customers in the system is equal to n, over a long 
period. 

The quantities of interest when we study a particular queueing system 
are above all the average number of customers in the system, when it is in 
equilibrium (or in stationary regime), and the average time that an arbitrary 
customer spends in the system. We introduce the following notations: 


N is the average number of customers in the system (in equilibrium); 
Ng is the average number of customers who are waiting in line; 


Ng is the average number of customers being served; 


T is the average time that an arbitrary customer spends in the system; 
Q is the average waiting time of an arbitrary customer; 


S is the average service time of an arbitrary customer. 


Often, we must content ourselves with expressing these quantities in terms 
of the limiting probabilities mp. Moreover, notice that N = No + Ng and 
T=Q+58. 

Let N(t), for t > 0, be the number of customers who arrive in the system 
in the interval [0, t]. Given that the random variables 7, are independent and 
identically distributed, the process {N(t),t > 0} is a renewal process. We 
denote by A, the average arrival rate of customers in the system. That is (see 
Prop. 5.6.7), 


(6.2) 





Remarks. i) If mn ~ Exp(A), then we obtain A, = À. 


ii) We can also define Àe, namely, the average entering rate of customers into 
the system. If all the arriving customers enter the system, then A, = Aa- 
However, if the capacity of the system is finite, or if some customers refuse 
to enter the system if it is too full when they arrive, etc., then A, will be 
smaller than Aa. Suppose, for example, that the customers arrive according 
to a Poisson process with rate A. If the system capacity is equal to c (<20) 
customers, then we may write that the rate A, is given by A (1 — re). Indeed, 
in this case, (1 — re) is the (limiting) probability that an arriving customer 
will enter the system. 


We can establish a relation between the quantities N and T by using a 
cost equation. Suppose that the customers entering the system pay a certain 
amount of money. Let A, be the average earning rate of the system. Then, 


318 6 Queueing Theory 


after a long enough time to, the average amount of money the system earns 
is approximately equal to A, - to. On the other hand, this quantity is also 
approximately equal to M-A,.-to, where M is the average amount of money 
a customer who enters the system pays, and 4, is defined above. The equality 
between the two expressions for the average amount of money the system 
earns, over a long period, can be justified rigorously. We then obtain the 
following proposition. 


Proposition 6.1.1. We have 
Ag = àe: M (6.3) 


Corollary 6.1.1. If the customers pay $1 per time unit that they spend in the 
system (waiting to be served or being served), then Eq. (6.3) becomes 


N=.-T (6.4) 


Remarks. i) The formula above is known as Little’? formula. 
ii) Little’s formula may be rewritten as follows: if t is large enough (for the 
process to be in equilibrium), then we have 


E[X(t)] = \.E(T| (6.5) 


where T is the total time that an arbitrary customer who enters the system will 
spend in this system. Actually, Eq. (6.5) is valid under very general conditions, 
in particular, for all the systems studied in this book, but in some cases it is 
not correct. Moreover, we can prove the following result: 


to 


lim — | X(t) dt = EIX (Ð) = MEIT] (6.6) 
0 


ty—oo to 


That is, the stochastic process {X(t),t > 0} is, in the cases of interest to us, 
mean ergodic (see p. 56). 


iii) If every customer pays $1 per time unit while being served (but not while 
she is waiting in queue) instead, then we obtain 


Ng = e- S (6.7) 
It follows that we also have 


No = .:Q (6.8) 


When the times between the arrivals of successive customers and the ser- 
vice times of customers are independent exponential random variables, the 


* John D.C. Little, professor at the Sloan School of Management of the Mas- 
sachusetts Institute of Technology, in the United States. 
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process {X(t),t > 0} is a continuous-time Markov chain. Moreover, if we as- 
sume (which will be the case, in general) that the customers arrive one at a 
time and are served one at a time, {X(t),t > 0} is then a birth and death 
process. We may therefore appeal to the results that were proved in Chapter 
3 concerning this type of process, particularly Theorem 3.3.4, which gives us 
the limiting probabilities of the process. 


6.2 Queues with a single server 


6.2.1 The model M/M/1 


We first consider a queueing system with a single server, in which the cus- 
tomers arrive according to a Poisson process with rate A, and the service times 
are independent exponential random variables, with mean equal to 1/1. We 
suppose that the system capacity is infinite, as well as the population from 
which the customers come. Finally, the queue discipline is that of first-come, 
first-served. We can therefore denote this model simply by M/M/1. 


The stochastic process {X(t),t > 0} is an irreducible birth and death 
process. We will calculate the limiting probabilities 7, as we did in Chapter 3. 
The balance equations of the system (see p. 140) are the following: 


state 7 departure rate from j = arrival rate to 7 


0 ATo = UTI 
n (> 1) (A + B)t = ATp—-1 + UTn41 


We have (see p. 141) 





Ig:=1 and HM - 4 = (2) for n = 1,2,... (6.9) 
HHH H 
es 
nx 


If A < u, the process {X(t),t > 0} is positive recurrent, and Theorem 3.3.4 
then enables us to write that 


FiO pete OR eee | 
Im A A 


Ins (>) (1 — =] Vn>0 (6.11) 
m m 


_ We can now calculate the quantities of interest. We already know that 
S = l/u. Moreover, because here Ae = A, we may write that 


That is, 
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À À À | 
H H H 


Fig. 6.1. State-transition diagram for the model M/M/1. 


Nes = : (6.12) 


Remarks. i) Since there is only one server, we could have directly calculated 
Ns as follows: 


Ñs=1-m=1- (1-2) =2 (6.13) 
u) B 

because the random variable Ns denoting the number of persons who are being 
served, when the system is in equilibrium, here has a Bernoulli distribution 
with parameter pọ := 1 — 7. | 

ii) The quantity p := A/u is sometimes called the traffic intensity of the sys- 
tem, or the utilization rate of the system. We see that the limiting probabilities 
exist if and only if p < 1, which is logical, because this parameter gives us the 
average number of arrivals in the system during a time period corresponding 
to the mean service time of an arbitrary customer. If p > 1, the length of the 
queue increases indefinitely. 


iii) From the balance equations of the system, we can draw the corresponding 
state transition diagram (and vice versa). Each possible state is represented by 
a circle, and the possible transitions between the states by arrows. Moreover, 
we indicate above or under each arrow the rate at which the corresponding 
transition takes place (see Fig. 6.1). 


Next, we have 


Eom (ECM OS 


n=l 


where Z ~ Geom (a — a) 


À u À 
ra — eee 6.14 
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We deduce from this formula that 





=. We A 
a ea (6.16) 
k Ne ulu- A) 
which implies that 
Lk À 1 1 
P2045 =. = 42S (6.17) 
" u(u-A) H pA 


Generally, it is difficult to find the exact distribution of the random variable 
T that designates the total time that an arriving customer will spend in the 
system (so that T = E[T}]). For the M/M/1 queue, we can explicitly determine 
this distribution. To do so, we need a result, which we will now prove. 


Proposition 6.2.1. Let &n be the probability that an arbitrary customer finds 
n customers in the system (in equilibrium) upon arrival. If the customers 
arrive according to a Poisson process, then we have 


Tn =A, Vn>0 (6.18) 


Proof. It suffices to use the fact that the Poisson process has independent 
increments. Suppose that the customer in question arrives at time t. Let Fe = 
the customer arrives in the interval [t,t + €). We have 


an = lim limP[|X(t") =n |F] = lim lim PHX) =n} N Fe. 


too €10 too €]0 P| F] 

Oopa p DE LXE) = n PIX (tE) =n] 

Eoo PIF] 

ind... PIF IP(X() =n] o a -e 

Se T a AAN 

= Jim XG jen Sm. 2 (6.19) 
OO 


Remark. It can also be shown that, for any system in which the customers 
arrive one at a time and are served one at a time, we have 


ORO, Vazo (6.20) 


where 6, is the long-term proportion of customers who leave behind them n 
customers in the system when they depart. This follows from the fact that 
the transition rate from n to n + 1 is equal to the transition rate from n + 1 
to n, over a long period. 
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Proposition 6.2.2. For an M/M/1 queue, we have T ~ Exp(p — A). 


Proof. Let R be the number of customers in the system when a new customer 
arrives. We have 


PIT < t] = S PIT <t|R=r]P|R =r] (6.21) 
r=0 


Moreover, by the memoryless property of the exponential distribution, we may 
write that 


T|{R=r}~G(r+1, u) (6.22) 


Finally, since the customers arrive according to a Poisson process, we have 
(by the preceding proposition) 


. AÀ 
PRST SOS ips (>) (: — >) (6.23) 
H H 
It follows that 


mar BU eEG CA 


~ fi u—dJe a (6.24) 


r=0 


(X 


Interchanging the summation and the integral, we obtain 


PT £i = fo — Aje ”s See = ds = -fu — je“ ^s ds (6.25) 
r=0 r! 
ee ee 


ers 


= fr(t)=(u—A)je“** fort >0 O (6.26) 


We can also calculate the distribution of the waiting time, Q, of an arbi- 
trary customer arriving in the system. This random variable is of mized type. 
Indeed, if the customer arrives while the system is empty, we have that Q = 0. 
On the other hand, if there are R = r > 1 customer(s) in the system upon his 
arrival, then Q has a G(r, p) distribution. Since P[R = 0] = rọ = 1—A/p, by 
proceeding as above, we find that 


PQ <t) = Y` PIR < t| R= r]P[R = r] (6.27) 
r=0 


OOO 
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Thus, we have P|Q = 0] = 1 — A/p and 
PIO<Q<t= (>) (1 — ae ift >0 (6.28) 
u 


Moreover, we calculate 


PIQ <t|Q>0) = Em E ift >0 (6.29) 


That is, Q | {Q > 0} ~ Exp(u — A). We may therefore write that 


OOS er (6.30) 


which follows directly from the fact that 


Tr a) Lee ie À 
Pinar|n>dje p = BUE a) ~) 
Sme SP R= Vea (6.31) 


Remark. Since the random variables Q and § are independent (by assump- 
tion), we can check that T has an exponential distribution with parameter 
jt — A by convoluting the probability density functions of Q and S. Because 
Q is a mixed-type random variable, it is actually easier to make use of the 
formula (with p = A/,1) 


oe) t 
Fr(t) = / Fo(t — s) fs{s) ds = J (1 — p g EA ue "s ds 


t 
=]—e#_ pue "| e** ds 
0 


=1—e Mt eH MEY — eM) 1 -e HM Yt>0 (6.32) 


The variance of the random variable N designating the number of cus- 
tomers present in the system in equilibrium (and whose mean value is N) is 
easily obtained by noticing [see Eq. (6.11)] that 


N +1 ~ Geom(1 — p) (6.33) 
It follows that 
Ap 


VIN] = V[N +1] = a7 = i (6.34) 


Note that the mean N can be expressed as follows: 


Nia ee (6.35) 
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In the case of the random variable Ng, we mentioned above that it has a 
Bernoulli distribution with parameter 1 — 7) = p. Then we directly have 


VINs] = p(t - p) = = (: ~ >) (6.36) 


To obtain the variance of Ng, we will use the following relation between 
Ng and N: 


6 fN=Oer1 M 
Bena C20) 
It follows that 
oO CO 
EIN] = XL Presa = 0? >) krk- = pL [Z| (6.38) 
k=1 k=1 


where Z ~ Geom(1 — p), from which we deduce that 


1 
E[N2] = e (aita) 6.39 
wal=e (aat Gop re 
Since the mean value of Ng |see Eq. (6.15)] is given by 


A2 p ) 
FTIN a 6.40 
No! ber) 1-p l 
we obtain that 


1+p A o PUTS) 
vna- a e) am a A 


Remark. Given that N = No + Ns, we can use the formula 
VIN] = V[Ng] + VINs] + 2 Cov[No, Ns] (6.42) 


to calculate the covariance (and then the correlation coefficient) of the random 
variables No and Ns. 


Now, we have shown that T ~ Exp(u — à). We then have 


VIT] = (6.43) 





Similarly, S ~ Exp(u) (by assumption), so that V[S] = 1/py?. Finally, by 
independence of the random variables Q and S, we obtain that 


_ \2 
1 fe 2urA — A (6.44) 


MQ) = VII VIS = ae a = uP 
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Remark. We can calculate V[Q] without making use of the independence of 
Q and S. We deduce indeed from the formula (6.30) that 


E|Q?] = E1Q? | Q > 0]PIQ > 0] = BIT*|(1 — 7o) = EIT] = r yya 
| (6.45) 
from which we obtain that 
vio = as eM (6.46) 


(u -A Wu- plu- A)? 


Example 6.2.1. Let K be the number of arrivals in the system during the 
service period of an arbitrary customer. We will calculate the distribution of 
K. First, we have i 

EFA 

where the service time S ~ Exp(p) and the time 7 needed for a customer to 
arrive, having an exponential Exp(A) distribution, are independent random 
variables. Then, by the memoryless property of the exponential distribution, 
we may write that 


k 
j p 
PRS for k = 1,2,... 
| ! ( A) (5) a | 


u 
K+1~Ge + 
com (5) 





PIK =0; =P 1S <7 = 








That is, 


from which we deduce that 
EtA) Ace 
u 


Thus, the average number of arrivals during the service period of a given 
customer is equal to the average number of arrivals in the course of a period 
corresponding to the mean service time of an arbitrary customer (see p. 320). 
This result is also easily proved as follows: 


E[K] = E|E[K | Sj] = EIAS] = AE[S] = = 


Example 6.2.2. The conditional distribution of the random variable N, namely, 
the number of customers in the system in stationary regime, knowing that 
N < m, is given by 


pr(n|N<m)= 2 pas A forms leem 


eo Te = Dopeo(A/H)A 
Note that the condition N < m does not mean that the system capacity is 
equal to m. It rather means that, at a given time instant, there were at most 
m customers in the system (in equilibrium). 
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Example 6.2.3. The queueing system M/M/1 is modified as follows: after hav- 
ing been served, an arbitrary customer returns to the end of the queue with 
probability p € (0, 1). 
Remark. A given customer may return any number of times to the end of the 
queue. | 
(a) (i) Write the balance equations of the system. 

(ii) Calculate, if they exist, the limiting probabilities 7;. 
(b) Calculate, in terms of the 7;’s (when they exist), 

(i) the average time that an arbitrary customer spends in the system, 

(ii) the variance of the number of customers in the system, given that it 
is not empty. 


Solution. (a) (i) The balance equations of the system are the following: 


state 7 departure rate from j = arrival rate to j 


0 Aro = a(l — p)m 
n (> 1) [A+ p(1 pnn = Ann-1 + uli — p)tn41 


(ii) The process considered is an irreducible birth and death process. We 


first calculate 
oà: = 
Si := 
Dooa Hi H2’ A E 


The infinite sum converges if and only if A < (1 — p). In this case, we find 
that 


À 
sy = —— 
' w(L—p)—X 
Then, the 7,’s exist and are given by 
v=o ara) = Gat) (aa) 
"ptt = p) (1 —p)—A (1 — p) (1 — p) 
0) ae ee bet Ge 


Remark. The result is obtained at once by noticing that the process considered 
is equivalent to an M/M/1 queueing system for which the service rate is 
(1 — p)p. 
(b) (i) We seek T. By the preceding remark, we may write (see the formula 
(6.17), p. 321) 

1 
pd =p) =) 


(ii) Let N be the number of customers in stationary regime. We have (see 
the formula (6.33), p. 323) 


T = 
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À 
N-+1~ Geom (1- -> ) 
a(l- p) 


Let M := N | {N > 0}. We seek V[M]. By the memoryless property of the 


geometric distribution, we can write that M ~ Geom( 1 — aati): It follows 


that 
i v[M) = ate | 0- at) = Au(1 — p) 
er a pL =p) - à}? 


6.2.2 The model M/M/1/c 


Although the M/M/1 queue is very useful to model various phenomena, it 
is more realistic to suppose that the system capacity is an integer c < oo. 
For j = 0,1,... ,c — 1, the balance equations of the system remain the same 
as when c = oo. However, when the system is in state c, it can only leave it 
because of the departure of the customer being served. In addition, this state 
can only be entered from c — 1, with the arrival of a new customer. We thus 


have 
state 7 departure rate from 7 = arrival rate to 7 


0 ATo = UTI 
Le a ae | (A + p)atR = ATR-1 + HUTk+1 
C Une = ATc-1 


The process {X (t), t > 0} remains a birth and death process. Moreover, given 
that the number of states is finite, the limiting probabilities exist regardless 
of the values the (positive) parameters \ and p take. 

As in the case when the system capacity is infinite, we find (see p. 319) 
that 


pN k 
Hp = (>) for k =0,1,...,¢ (6.47) 


It follows, if p := A/pu #1, that 


c+1 


k c 
Dm=) (3) = pt = (6.48) 


k=0 k=0 


When A = u, we have that p = 1, Hp = 1, and $`% _—o MHk = c + 1, from which 
we calculate 
el =p) 
I; I= 1a ptt pre 


k=0 


Tj = 





tos] 
c+ 1 1 p 


bra =U ee 
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Remarks. i) We see that if A = u, then the c+ 1 possible states of the sys- 
tem in equilibrium are equally likely. Moreover, when c tends to infinity, the 
probability m; decreases to 0, for every finite j. This confirms the fact that, 
in the M/M/1/00 model, the queue length increases indefinitely if A = p, so 
that there is no stationary regime. 

ii) If A > u, the limiting probabilities exist. However, the larger the ratio p = 
A/ is, the more re increases to 1 (and 7; decreases to 0, for j = 0,1,... ,e—1), 
which is logical. 

iii) Even if, in practice, the capacity c cannot be infinite, the M/M/1 model 
is a good approximation of reality if the probability 7, that the system is full 
is very small. 


With the help of the formula (6.49), we can calculate the value of N. 


Proposition 6.2.3. In the case of the M/M/1/c queue, the average number 
of customers in the system in equilibrium is given by 


c+1 c+i 
Be OD EE 








Nellan 1- pe (6.50) 
c/2 ifp=1 
Proof. First, when p = 1, we have 
t= 1 e(c+1) c 
N= = ^ = 6.51 
Si CF == e+1 ori i c+1 2 2 Pan 


k=0 


When p # 1, we must evaluate the finite sum 


pe ren 1 A ok 
N= Di I Tapert Lhe (1 — p) (6.52) 


Let X := Z — 1, where Z has a geometric distribution with parameter 1 — p. 
We find that the probability mass function of X is given by 
px(k)=(1—p) fork =0,1,... (6.53) 


It follows that 


CO OO 
1 p 
k ae — k — ee e — 54 
> p)=1 and >» kp" (1 a ha (6.54) 


Making use of these formulas, we may write that 


Dera- —- 5 kpt (1 — p) (6.55) 


k=c+1 
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and 
OO OO 
>, ke =p) = So fk-(c+1) + (C+D) - p) 
k=c+1 k=c+1 
OO OO 
= Ý kk- (e+ E0- Y (c+ D*A- p) 
k=c+1 k=c+1 
OO OQ 
=p X mQ- p) + (e+ 1p! Y a- p) 
— pt- 4 (c + per (6.56) 
l — p 
so that 
5 1 p p i 
Ne ef] FO 1 e+ 
Saher i a 
1 p 
7 c+l c+1 
= {l-e (15) = (c+ 1)p \ O (6.57) 
Remarks. i) We easily find that 
| ra 
z fp<l 
imNes top (6.58) 
ATE œ ifp>l 


which corresponds to the results obtained in the preceding subsection. 


ii) When the capacity c of the system is very limited (for example, when 
c = 2,3, or 4), once we have calculated the 7;’s, we can directly obtain N 
from the definition of the mathematical expectation of a discrete random 
variable: N := Yok Tpk. We can then also calculate the variance of the 
random variable N as follows: 


VIN] := X (k - E[N] rr = X k?ng — (EIN)? (6.59) 
k=0 k=0 


iii) When p = 1, we find, using the formula 


NK = Acter (6.60) 
k=0 
that 
c c 
1 ce(2c + 1) 
EIN?) := “th =) kh? = 6.61 
ale ee 
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so that 


v= S220 (gD ay 


Now, as in the case of the M/M/1/œ queue, we have 


c 
Ns =) 1-1 =1-— To (6.63) 
k=1 
and 
Ng =N—1+70 (6.64) 


Next, the average entering rate of customers into the system (in equilib- 
rium) is given by 


Ae = A(1 — Te) (6.65) 


because the customers always arrive according to a Poisson process with rate 
A but can enter the system only if it is not full (or saturated), that is, if it is 
in one the following states: 0,1,...,c — 1. Using both this fact and Little’s 
formula (see p. 318), we may write that the average time that a customer 
entering the system spends in this system is equal to 


N 


I E (6.66 
For the customers entering the system, we still have that S = 1/p (by 


assumption). Then 


- V 1 

ae ae (6.67) 
AQ rT) pb 

Remark. If we consider an arbitrary customer arriving in the system, the 

average time that she will spend in it is then T = N/A, as noted previously. 

Indeed, in this case the random variable T is of mixed type, and we may write 

that 


E(T] = EJT | T = 0} P|T = 0] + E[ | T > 0}PIT > 0] 


(=r) = 2 (6.68) 


= 0X re + ài 
7 ° " X(1 — te) 


We also have 


(1 — re) (6.69) 


| = 


S= (1-m) = gs 


= a 
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Example 6.2.4. We consider a queueing system with a single server and finite 
capacity c = 3, in which customers arrive according to a Poisson process with 
rate À and the service times are independent exponential random variables, 
with parameter u. When the system is full, the customer who arrived last will 
be served before the one standing in line in front of him. 


The balance equations of the system are the following: 


state 7 departure rate from 7 = arrival rate to 7 


Ato = UTI 
(A + u)r = Ato + T2 
(A + u)r2 = Àn + UT3 
prs = AT 


WN kr © 


Notice that these equations are the same as the ones obtained in the case of the 
M/M/1/3 system, although the queue discipline is not the default one (that 
is, first-come, first-served). When = A, we know that the solution of these 
equations, under the condition Y`? <0 Tj = 1, is m; = 1/4, for j = 0,1, 2,3. 

Suppose that a customer arrives and finds exactly one person in the system. 
We will calculate the mathematical expectation of the total time T that this 
new customer will spend in the system if u = A. Let K be the number of 
customers who will arrive after the customer in question but will be served 
before her (if the case may be). By the memoryless property of the exponential 
distribution, we may write that 


K +1 ~ Geom(1/2) 


It follows that 


Me 


E[T;] = 


a 
Il 


0 


EIT, | K = KPI ge ety 
k=0 
ores l ) k+1 
2 
ae al 


k=0 
1 1/1 _,\_3 
s(a) 


Finally, let Ts be the total time that an entering customer will spend in the 
system, and let L be the number of customers already present in the system 
upon his arrival. We may write that 


NO] m= 


| 
>| th 
ig 


0 


| dS 


z SE |L =ĮPIL = 
i=0 
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Given that (in stationary regime) 


AS ae eat 
PIN <2] 3/4 3 


we then deduce from what precedes that 


KIE 23). ON. 2 
ein)" 3 (5+5 +5) =5 


PIL=l]|=PIN=I|N<2|= 


3B3\A A AJ A 


Notice that when L = 2, it is as if the customer we are interested in had 
arrived at the moment when there was exactly one customer in an M/M/1/3 
system with service policy FIFO (first-in, first-out). 


6.3 Queues with many servers 


6.3.1 The model M/M/s 


An important generalization of the M/M/1 model is obtained by supposing 
that there are s servers in the system and that they all serve at an exponential 
rate u. The other basic assumptions that were made in the description of 
the M/M/1 model remain valid. Thus, the customers arrive in the system 
according to a Poisson process with rate A. The capacity of the system is 
infinite, and the service policy is that by default, namely, first-come, first- 
served. 

We suppose that the arriving customers form a single queue and that the 
customer at the front of the queue advances to the first server who becomes 
available. A system with this waiting discipline is clearly more efficient than 
one in which there is a queue in front of each server, since, in this case, there 
could be one or more idle servers while some customers are waiting in line 
before other servers. 


Remark. A real waiting line in which the customers stand one behind the other 
need not be formed. It suffices that the customers arriving in the system take 
a number, or that the tasks to be accomplished by the servers be numbered 
according to their arrival order in the system. 


Since the customers arrive one at a time and are served one at a time, 
the process {X(t),¢t > 0}, where X(t) represents the number of customers in 
the system at time t, is a birth and death process. Note that two arbitrary 
customers cannot leave the system exactly at the same time instant, because 
the service times are continuous random variables. The balance equations of 
the system are the following (see Fig. 6.2 for the M/M/2 model): 


state 7 departure rate from 7 = arrival rate to 7 
0 ATO = UTI 


O<k<s (A+ kung = (k + L)Unky1 + AtK-1 
k>s (A + sp)arp = SUTk+1 + ÀTk-1 
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N A A 
u 2p 2u 


Fig. 6.2. State-transition diagram for the model M/M/2. 


We can solve this system of equations under the condition J ponk = 1. As 
in the case of the M/M/1 model, it is, however, simpler to use Theorem 3.3.4. 


We first calculate the quantities Hg, for k = 1,2,.... The cases when k < s 
and when k > s must be considered separately. We have 
_jkpiflek<s 
Oe Perea oe) 
Then 


AXAR EGRA 1 fA 
k= ux2ux- -xku kl 


k 
>) fork =(0), 1, 2e (6.71) 


n 


and, for k =s +1,s+2,..., 


NEN Sane ete t poy 
= Eee i (ec (6.72) 
UXBwWree-Xsuxsux---xX su sisk-s \ u 
S re 
({k—s) times 


Given that the stochastic process {x (t), t > 0O} is irreducible and that 


= À 
S:= >) I,<00 = pia =e (6.73) 
k=l 


we can indeed appeal to Theorem 3.3.4. We first have [see Eq. (3.301)| 


8 k s © k = so) o s? = P ‘ - 
iea FO) - EA SEO 
1 k=s+1 k=0 ad 


1 


s—-i ķ s -1 
p<s P P 5 
= Oa, ee eee 6.74 
enn ee 
=0 
Once 7 has been calculated, we set 
k 
To a E O 
Wk = i (6.75) 
k 


-L m fk =s+1,8+2,... 
so 
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We will now calculate, in terms of ro, the average number of waiting 
customers in the system in stationary regime. We have 


2 ieee saat ro= Sm (k-s)(2) 


k=st+1 k=s+1 k=s+1 
(6.76) 
Next, 
OO OO ; 
E ue) Geer E E e eat tae Peeters: 
2 i s) (5) (>) 2a E ) ( J 1 — (p/s) 
k=s+1 j=l 
(6.77) 
from which we obtain | 
7 peti 92 peti 1 
= ee eee eee 6.78 
No sis (s— pe” sis (1- g)2" oe) 
where € := p/s. We can also write that 
= 1 £ 
ee o ne 6.79 
No= (rept = Gee nm 


From No and Little’s formula, we deduce all the other quantities of inter- 
est. Since the average entering rate of customers into the system is Àe = À, 
we may write that 

= Pe i 
z Q a p? S 
= E E E E 6.80 
ae Ns! (s—p)? ° u (9:90) 


because S = 1/ for every server (by assumption). 
Finally, we have 


Ng =AS=p and N=AT=No+p (6.81) 


Remarks. i) We can also calculate, in particular, the probability 7, that all 
the servers are busy. We have 


ok s? on pyk 
= Smad gaa = God (E) 
=F 


k>s k=s 
pcs s? ey 1 _ sp? 6.82 
s!°\s) 1-0/9 se- a 


ii) If the number s of servers tends to infinity, then we find that 


k 
To — eò and Tmk — O et TOP ac 2 ee (6.83) 


6.3 Queues with many servers 335 

That is, in the case of the M//M/oo model, we have 
nk = P|Y =k], where Y ~ Poi(A/p) (6.84) 
Since No =Q=0 (because there is no waiting time), we then directly obtain 


Ñ = BY] = 2 = Ñs and T=5=- (6.85) 


As we did for the M/M/1 model, we can explicitly find the distribution of 
the random variable Q, namely, the time that an arbitrary customer spends 
waiting in line. This variable is of mixed type. We have 


P[Q =0) = 1 — 7 (6.86) 
and, for t > 0, 
PO0<Q<t=) POLO stiker (6.87) 


where R is the number of customers in the system upon the arrival of the 
customer of interest. When at least s customers are in the system, the time 
needed for some customer to depart is a random variable having an exponen- 
tial distribution with parameter su. It follows that 


Q|{R=r}~G(r—s4+1,su) forr=s,s+1,... (6.88) 


because the customer must wait until r — s + 1 persons ahead of him have left 
the system before starting to be served. Making use of the fact that 


P[R =r] =, (6.89) 


(because the arrivals constitute a Poisson process) and proceeding as in the 
case when s = 1, we find that 


FOS O <7) =, (1 — ene) for t > 0 (6.90) 


Finally, adding the probabilities P[Q = 0] and P[0 < Q < t], we obtain the 
following proposition. 


Proposition 6.3.1. The distribution function of the random variable Q in an 
M/M/s queueing system (for which A < su) is given by 


Fo(t) = P[Q <t]=1—meO-* fort >0 (6.91) 
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Remarks. i) With s = 1, we calculate 


=j 
tass (i + =) =p (6.92) 
1—p Lop 


so that 
PIQ <t] = 1-pe0-™* fort >0 (6.93) 


which corresponds to the formula (6.27). 


ii) To obtain the distribution of the total time, T, that an arbitrary customer 
spends in an M/M/s system in equilibrium, we would have to compute the 
convolution product of the probability density functions of the random vari- 
ables Q and S ~ Exp(y). 


Example 6.3.1. The number K of idle servers in an M/M/s system, in sta- 
tionary regime, may be expressed as follows: 


K =s- Ns 


where Ns is the number of customers being served. We then deduce from Eq. 
(6.81) that i 
E|K| =s- Ns =s8s-p 


6.3.2 The model M/M/s/c and loss systems 


As we mentioned in Subsection 6.2.2, in reality the capacity c of a waiting 
system is generally finite. In the case of the M/M/s/c model, we can use the 
results on birth and death processes to calculate the limiting probabilities of 
the process. 


Example 6.3.2. Consider the queueing system M/M/2/3 for which À = 2 and 
u = 4. The balance equations of the system are 


state 7 departure rate from 7 = arrival rate to 7 


270 ® Atty 


(2 + A)\ry © 279 + (2 x A) 
(242% 4ra 2 2m1 + (2 x 4)r3 
(2 x A)irg 8) 272 


QW no e © 


We can directly solve this system of linear equations. Equation (0) yields 
Ti = iro. Substituting into (1), we obtain 72 = in = ino. Next, we deduce 


from Eq. (3) that a3 = in = A To. We can then write that 
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ny T T: =1 = a = po ee 
A ale we A R G3 


Note that this solution also satisfies Eq. (2). 
The average number of customers in the system in equilibrium is given by 


3 
p 16 4 1 OF 
= Z eo — 3 —_ = 
A D 53 T2 X g3 + OX pg = Bg 


from which the average time that an entering customer spends in the system 
is o 

‘ 
ee Os gene 
Ae 2(1— 7m3) 2(52/53) 104 


We also find, in particular, that the limiting probability that the system is 
not empty is given by 1 — 7 = #. 


T= 


Particular case: The model M/M/s/s 


When the capacity of the system is equal to the number of servers in the 
system, no waiting line is formed. The customers who arrive and find all the 
servers busy do not enter the system and are thus lost. Such a system is called 
a no-wait system or a loss system. 

Let X(t), for t > 0, be the number of customers in the queueing system 
M/M/s/s at time t. We find that the balance equations of the system are 
given by 


state j departure rate from j = arrival rate to 7 


0 Ato = [LT 
O0<k<s (A + kurg = (k +1) Tk41 +ÀTk-1 
8 SEL 1%, == ATs—-1 


The birth and death process {X (t), t > 0} is irreducible and, since 


IT, = 


AXAX+++ XX a À 
xx -xku k! 


k 
>) for k= 125048 (6.94) 
H 


the sum Sı is given by 


8 s 1 À k 
s=) m=) g>) (6.95) 
k=1 ` 


L 
k=1 I 


This sum is finite for all (positive) values of À and u. Theorem 3.3.4 then 
enables us to write that 


p"/k! 


5 a for. c= i hae S (6.96) 
j=0 ' 


Tk = 
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where p := A/p. Thus, the z,’s correspond to the probabilities P[Y = k] of 
a random variable Y having a truncated Poisson distribution. Indeed, if Y ~ 
Poi(p), then we have 


e~P oF /k! p* /k! 


PS a Song PTR 


for k = 0,1,...,8 
(6.97) 


Note that in the particular case when à = p, the limiting probabilities 7, 
do not depend on X (or py: 


. Wk! 


Moreover, in the general case, the probability mẹ that all servers are busy 
is simply 


À 
Tk 


for k= 0.152058 (6.98) 


p° fs! 
This formula is known as Erlang’s formula. 


Finally, the average entering rate of customers into the system is Àe = 
A(1 — rs). It follows that 


= RF for all p > 0 (6.99) 


5 7 1 

N=,A,T=X1 - a) (6.100) 
Since there is no waiting period in this system, as in the M/M/oo model, we 
have that N = Ns, T = S, and Ng = Q =0. 
Remarks. i) If the number of servers tends to infinity, we obtain 


$ 
lim X p/j! =e = e (6.101) 
=0 


$—00 4 


and we retrieve the formula (6.83): 


k 
lim mk = OV -Mu lotka 0; 1st (6.102) 


S— CO 
ii) Note that the limiting probabilities may be expressed as follows: 
___ ESI)" /k! 
X= (AE[S])’ /J! 
Now, it can be shown that the formula (6.103) is also valid for the more general 
model M/G/s/s (known as Erlang’s loss system), in which the service time 
is an arbitrary nonnegative random variable. This result is very interesting, 


because it enables us to treat problems for which the service time does not 
have an exponential distribution. 


for k = 0,1,...,8 (6.103) 
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Example 6.3.3. We consider the queueing system M/G/2/2 in which the ar- 
rivals constitute a Poisson process with rate À = 2 per hour. 


(a) What is the probability 79 that there are no customers in the system at 
a (large enough) time t if we suppose that the service time S (in hours) is an 
exponential random variable with mean equal to 1/4? What is the average 
number of customers in the system in equilibrium? 


(b) Calculate the value of 79 if S is a continuous random variable whose 
probability density function is 


fs(s) = 64se~°* for s > 0 


Solution. (a) When S ~ Exp(y = 4), we have the M/M/2/2 model with 
p = 2/4 = 1/2. We seek 


1 1 8 


Toa = —— OF m Z Ã— 
"2 (1/27/71 1+4+} 13 


We also find that 


E es nd s ne 
Pe ge ra, Serene T 


It follows that : i í 5 

N=Ox TIX Tan 13 
(b) We can check that the density function above is that of a random variable 
having a gamma distribution with parameters a = 2 and à* = 8. It follows 
that E[S] = a/A* = 1/4. Since the mean of S is equal to that of a random 
variable having an Exp(4) distribution, we can conclude that the value of 
To is the same as that in (a). Actually, we can assert that all the limiting 
probabilities 7, are the same as the z,’s in (a). Consequently, we have that 
N = 6/13 as well. If we do not recognize the distribution of the service 
time S, then we must calculate the mean EJS] by integrating by parts, or by 
proceeding as follows: 


CO OO 
ES | is | s- 64se™"ds = | 64s" Sds 
0 0 


t=8s 1 ss 2 _¢ 1 2! 1 
= = te "dt = -r(3) = — = - 
8 | UE EE E 
Remark. The queueing systems M/M/s (= M/M/s/œ) and M/M/s/s are 
the two extreme cases that can be considered. The model M/M/s/c, with 
s < c< œ, is the one that can most often represent reality well, because there 


is generally some space where potential customers can wait until being served, 
but this space is not infinite. If the capacity of a queueing system is finite, 
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and if this system is part of a network of queues (see the next subsection), 
then we say that the possibility of blocking of the network exists. 

Moreover, we found that for the system M/M/s to attain a stationary 
regime, the condition A < su must be satisfied. That is, the arrival rate of 
customers in the system must be smaller than the rate at which the customers 
are served when the s servers are busy. Otherwise, the length of the queue 
increases indefinitely. However, in practice, some arriving customers will not 
enter the system if they deem that the queue length is too long. Some may 
also decide to leave the system before having been served if they consider that 
they have already spent too much time waiting in line. 

To make the model M/M/s more realistic, we may therefore suppose that 
the customers are impatient. A first possibility is to arrange things so that 
the probability that an arriving customer decides to stay in the system and 
wait for her turn depends on the queue length upon her arrival. For example, 
in Exercise no. 4, p. 346, we suppose that the probability r, that an arriv- 
ing customer who finds n persons in the system decides to stay is given by 
1/(n + 1). This assumption leads to a particularly simple solution. We could 
also suppose that rn = K”, where x € (0, 1], etc. 

When the potential customers decide by themselves not to enter the sys- 
tem, we speak of a priori impatience. We call a posteriori impatience the case 
when the customers, once entered into the system, decide to leave before hav- 
ing been served, or even before their service is completed. This situation may 
be expressed as follows: an arbitrary customer, who entered the system, de- 
cides to leave it if Q > to or if T > tı, where tọ and tı are constants fixed 
in advance. We can also imagine that the time that an arbitrary customer is 
willing to spend in the system (or waiting in line) is a random variable having 
a given distribution (an exponential distribution, for example). 


Example 6.3.4. (a) Suppose that, in the loss system M/M/K/K, for which 
A = pt, the quantity K is a random variable such that 


K= 1 with probability 1/2 
-~ | 2 with probability 1/2 


(i) Calculate the average number of customers in the system in stationary 
regime. 

(ii) For which value of K is the average profit (per time unit) larger if each 
customer pays $x per time unit and each server costs $y per time unit? 


(b) Redo question (i) of part (a) if K is instead a random variable such that 
P\K = k| = 1/8, for k = 1, 2, and 3, and if the capacity of the system is 
c = 3, so that potential customers can wait to be served (when K = 1 or 2). 


Solution. (a) (i) If K = 1, we have (see the formula (6.96) or (6.98)): 


1 
i+ 


A= 
A= 
SH 
e—a 








To = and Tı = = 


1 
2 


ae 
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so that E[N | K = 1} = 1/2. When K = 2, we find that 


1 =p 2 1 
9 = Cl 71 =o, and magna 


2 ’ 
ee oa ae o 
1+ reg (>) 5 
It follows that E[N | K = 2] = 4/5. Finally, we have 


> 1 13 
N= E[N] = 5{E[N | K = 1] + E[N | K = 2p = 5 
(ii) If K = 1, the average profit per time unit, Pr, is given by $ ($2 — y), 
while Pr = $ ($x — 2 y) when K = 2. Therefore, the value K = 1 is the one 
for which the average profit is larger if and only if 


1 


4 3 
la ae a <> ¥> 75% 


(b) First, if K = 1 (and c = 3), we have (see the formula (6.49)) m; = 1/4, for 
i = 0,1,2,3. Then E[N | K = 1] = 3/2. Next, if K = 2, we write the balance 
equations of the system (see Ex. 6.3.2): 


state 7 departure rate from 7 = arrival rate to 7 


AT 2 HTI 
2 
(A+p)m È Aro + 2uT2 
3 
(A + 2u) 2 Amy + 2UT3 


Ww bw e O 


273 © AT 


When A = u, we find that (1) implies that mọ = 71. Furthermore, (2) then 
implies that m2 = 7/2. It follows, from (4), that m3 = 7/4. Making use of 
the condition 7 + 7, + 72 +73 = 1, we obtain that 


ty =m == m= =, and maa = EIN|KkK=2])=1 
Finally, with K = 3, we calculate 
a eee 5 ae Ki = KO; cas apes and fae ae 
rene) eG) a ™ l 
so that 
I 15 
BIN | K =3)=0x Č +1x 7 +2x 5+3 


The average number of customers in the system (in stationary regime) is 


therefore 


1/3 15 59 
Ni=-(- — | = — ~ 1.146 
HN 3 € es =) 48 
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6.3.3 Networks of queues 


We consider a network made up of k queueing systems. In the ith system, 
there are s; servers (each of them serving only one customer at a time), for 
i = 1,2,...,k. We suppose that the capacity of each system is infinite and 
that customers, coming from outside the network, arrive in system 7 according 
to a Poisson process with rate 8;. The k Poisson processes are independent. 

After having left the queueing system i, an arbitrary customer goes to 
system j € {1,2,...,k} with probability p;;, so that the probability that the 
customer leaves the network (after having been served in system 7) is given 
by | 


k 


pio =1-J pij 20 (6.104) 
j=l 


We assume that the probability that a given customer remains indefinitely 
in the network is equal to zero. Finally, the service times are independent 
exponential random variables with rates y;, for i = 1,...,k, and are also 
independent of the times between the successive arrivals. 


Remarks. i) A network of this type is said to be open, because the customers 
can enter and leave the system. We could also consider the case when the 
network is closed, that is, the number of customers is constant, and they 
move indefinitely inside the network. 


ii) Notice that the probability p;,; may be strictly positive, for any i. That is, 
it is possible that a customer, after having departed system 7, returns to this 
system immediately. 


Let X(t) := (X1(t),...,Xx(t)), where X;(t) designates the number of 
customers in system 7? at time t, for i = 1,...,k. We want to obtain the 
distribution of X(t) in stationary regime. Let A; be the total rate at which 
customers arrive in system j. Since the arrival rate into a queueing system 
must be equal to the departure rate from this system, the A;’s are the solution 
of the system of equations 


k 
Ay =O; +302 Pig TOI =l; 2 esik (6.105) 
i=1 
Once this system has been solved, the next theorem, known as Jackson’s? 


theorem, gives us the solution to our problem. 


Theorem 6.3.1. Let Ni = lity X;(t). If Ài < Silti; fori = 1, wee a oF then 


k 
Jim PX =nl =P N {N; = na (6.106) 
ix=1 


3 James R. Jackson, emeritus professor at UCLA (University of California, Los 
Angeles), in the United States. 
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where n := (n1,... nk), and N; is the number of customers in an M/M/s; 
queueing system in stationary regime. 


Remarks. i) Under the assumption that all customers eventually leave the 
network, it can be shown that the system (6.105) has a unique solution. 


ii) The statement of the theorem is surprising, because it implies that the 
random variables N; are independent. If an arbitrary customer cannot return 
to a system he already departed, then the arrivals in each system constitute 
Poisson processes, because the departure process of an M/M/s queue is (in 
stationary regime) a Poisson process with rate À (if A < sy). Moreover, these 
Poisson processes are independent. In this case, the result of the theorem is 
easily proved. However, when a customer may be in the same system more 
than once, it can be shown that the arrival processes are no longer Pois- 
son processes. Indeed, then the increments of these processes are no longer 
independent. Now, according to the theorem, the random variables are nev- 
ertheless independent. 

iii) When there is a single server per system, so that s; = 1, the network 


described above is called a Jackson network. We then deduce from the formula 
(6.11) that 


k NNi l 
fim, PIO X) = (mew = TE (ZE) (2) eon 


(if A; < yj, for all i). The average number of customers in the network in 
equilibrium is then given by 





k 
sla Fd Pires (6.108) 


, i i 
į=Í1 i 


Finally, given that the average entering rate of customers (coming from the 
outside) into the network is 


TD (6.109) 


i=] 


the average time that an arbitrary customer spends in the network is 


Tao eae = (6.110) 


Example 6.3.5. The simplest example of a Jackson network is that of a sequen- 
tial system in which there are two servers and the arriving customers must 
necessarily go the first server, and next directly to the second one. Then they 
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leave the system. We suppose that the first queueing system is an M/M/1 
model, with service rate 1. Thus, the arrivals in the network constitute a 
Poisson process with rate \. Similarly, we suppose that the capacity of the 
second queueing system is infinite and that the server performs the desired 
service (to only one customer at a time) at an exponential rate u2. Finally, 
all the service times and the times between the successive arrivals are inde- 
pendent random variables. 

Since the customers cannot find themselves more than once in front of the 
same server, we can assert, if A < u1, that the departure process of the first 
system (in stationary regime) is also a Poisson process with rate A, so that 
the second queueing system is an M/M/1 model as well. We deduce from 
Jackson’s theorem that 


Tana = lim P[(X1(t), Xo(t)) = (m, n2)] 


AOA OA) om 


for nı and ng € {0,1,...}. 

We can show that the formula above is valid by checking that the joint 
limiting probabilities 7, n, satisfy the balance equations of the network. Sup- 
pose, to simplify further, that 4; = uo := u. These balance equations are then 


state (7,7) departure rate from (i, 7) = arrival rate to (i, 7) 


(0, 0) AT0,0 = HTO, 
(71,0), nı > 0 (A T 4)Tn;,0 = Unn, + ÀTn;—1,0 
(0, n2), n2 >0 (A + 1) 70,n2 = U(To,n2+1 a Tinz=1) 
(n1, n2), nin > 0 (A + 21) Trine = H(A not1 + Mn 41,n2-1) 
+ARn,—-1,n2 


When u1 = 42 = u, we can rewrite the formula (6.111) as follows: 


Ny+tn2 2 
ias (=) h — >) (6.112) 
u 7 


We have, in particular, 


(A T 2u)T ny, nz T H{Tni n+ + Tn1+1,n2—-1) F AT ny —1,n2 
= 


Nitne nitnetl Ni tne my—l+ne © 
aaa (3) ad (AY BY] G) 
p H H p 
=> 


=i 
A+2u=n|(2) 41] +a (4) =A\+ pty (6.113) 


6.3 Queues with many servers 345 


By the uniqueness of the solution, under the condition eae) Mts tig ds 
we can then conclude that the probabilities mn; n, are indeed given by the 
formula (6.111). 


Remarks. i) When ty = u2 = pm, the limiting probabilities t,,, depend 
only on the sum nı + n2. However, the probability 7, n, is not equal to the 
probability that there will be exactly nı + ng customers in the network in 
stationary regime. Indeed, we calculate, for example, 


jim P[Xi(t) + X(t) = 1] ™=" 2 lim PX (t) = 1, Xo(t) = 0) 


2 
= (>) (i P *) (6.114) 
H H 
Moreover, we have 


oo À nytne X 2 N 2 Oo À nitne 
E T TN 


nı +nz=0 
2 
E: E tee a (6.115) 
À m 





u aT 
while 
Oo 
>) fim PIM @) + Xo) = m1 +2] = 1 (6.116) 
nytn2=0 


ii) We can also write that 


Ti = Jim P[X1(t) = nı + na, X2(t) = 0) 


= lim P[X1(t) = 0, X2(t) = mı + n2] (6.117) 
More generally, we have 
Tnm = lim P[Xi(t) = i, X2(t) = j] (6.118) 
00 


for all nonnegative integers i and j such that i+ j = ny + ng. Thus, when 
[41 = u2, all the possible distributions of the nı + n2 customers between the 
two servers are equally likely. 


iii) Finally, the network described in this example is different from the M/M/2 
model, even if the two service rates are equal. Indeed, when there are exactly 
two customers in this network, both these customers may stand in front of 
server 1 (or server 2), whereas, in the case of the M/M/2 model, there must 
be one customer in front of each server. 
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6.4 Exercises 


Remark. In the following exercises, we assume that the service times are ran- 
dom variables that are independent among themselves and are independent 
of the times between successive arrivals. 


Section 6.2 


Question no. 1 
Calculate the average number of arrivals in the system during the service 
period of an arbitrary customer for the queueing model M/M/1/3. 


Question no. 2 

Drivers stop to fill up their cars at a service station according to a Poisson 
process with rate A = 15 per hour. The service station has only one gasoline 
pump, and there is room for only two waiting cars. We suppose that the 
average service time is equal to two minutes. 


(a) Calculate N and No. Why is N 4 No +1? 
Q Q 
(b) If we suppose that an arriving driver who finds the three spaces occupied 


will go to another service station, what proportion of potential customers is 
lost? 


Question no. 3 

Airplanes arrive at an airport having a single runway according to a Pois- 
son process with rate À = 18 per hour. The time during which the runway is 
used by a landing airplane has an exponential distribution with mean equal 
to two minutes (from the moment it receives the landing authorization). 


(a) Knowing that there is at most one airplane in the system (at a given time 
instant), what is the probability that an arriving airplane will have to wait 
before being allowed to land? 


(b) Given that an airplane has been waiting for the authorization to land for 
the last 5 minutes, what is the probability that it will have landed and cleared 
the runway in the next 10 minutes? 


Question no. 4 

We suppose that the probability that an arriving customer in an M /M/1 
queueing system decides to stay and wait until being served is given by 
1/(n + 1), where n is the number of customers in the system at the time 
when the customer in question arrives, for n = 0,1,2,. 


(a) Calculate N and No. 


(b) What is the percentage of customers who decide not to enter the system? 


Question no. 5 

We wish to compare two maintenance policies for the airplanes of a certain 
airline company. In the case of policy A (respectively, B), the airplanes arrive 
to the maintenance shop according to a Poisson process with rate A4 = 1 
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(resp., Ag = 1/4) per day. Moreover, when policy A (resp., B) is used, the 
service time (in days) is an exponential random variable with parameter u4 = 
2 (resp., the sum of four independent exponential random variables, each of 
them with parameter upg = 2). In both cases, maintenance work is performed 
on only one airplane at a time. 


(a) What is the better policy? Justify your answer by calculating the average 
number of airplanes in the maintenance shop (in stationary regime) in each 
case. 


Indication. The average number of customers in a queueing system M/G/1 
(after a long enough time) is given by 


Sa X2 E[S?] 
Ñ = XE[S] + AEST 


where S is the service time and À is the average arrival rate of customers. 


(b) Let N be the number of airplanes in the maintenance shop in stationary 
regime. Calculate the distribution of N if policy A is used, given that there 
are two or three airplanes in the shop (at a particular time instant). 


(c) If policy A is used, what is the average time that an airplane, which has 
already been in the maintenance shop for two days, will spend in the shop 
overall? 


Question no. 6 

We consider a queueing system in which there are two types of customers, 
both types arriving according to a Poisson process with rate A. The customers 
of type I always enter the system. However, the type II customers only en- 
ter the system if there is no more than one customer in the system when 
they arrive. There is a single server and the service time has an exponential 
distribution with parameter pL. 


(a) Write the balance equations of the system. 


(b) Calculate the limiting probability that an arbitrary type II customer enters 
the system if A = 1 and u = 2. 


Indication. The system considered is a birth and death process. 


(c) Calculate the average time that a given arriving customer of type H will 
spend in the system if A = 1 and p = 2. 


Question no. 7 

We consider a waiting system with a single server and finite capacity c = 
3, in which the customers arrive according to a Poisson process with rate 
A and the service times are independent exponential random variables with 
parameter u = 2\. When the system is full, a fair coin is tossed to determine 
whether the second or third customer will be the next one to be served. 


(a) Write the balance equations for this system, and calculate the limiting 
probabilities 7;. 
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(b) Calculate the average time that an arriving customer who finds (exactly) 
one customer in the system will spend in it. 


(c) Calculate the average time that a customer who enters the system will 
spend in it. 


Question no. 8 

Customers arrive at a service facility according to a Poisson process with 
rate A = 10 per hour. The (only) server is able to serve up to three customers 
at a time. The service time (in hours) has an exponential distribution with 
parameter p = 5, regardless of the number of customers (namely 1, 2, or 3) 
being served at the same time. However, an arbitrary customer is not served 
immediately if the server is busy upon her arrival. Moreover, we suppose that 
the service times are independent random variables and that there can be at 
most three customers waiting at any time. We define the states 


0’: nobody is being served 
0: the server is busy; nobody is waiting 
n: there is (are) n customer(s) waiting, for n = 1,2,3 


(a) Write the balance equations of the system. 
(b) Calculate the limiting probabilities 7, for all states 7. 
(c) What is the probability that an arriving customer will be served alone? 


Question no. 9 (Modification of the preceding question) 

Customers arrive at a service facility according to a Poisson process with 
rate À. The (only) server is able to serve up to two customers at a time. 
However, an arbitrary customer is not served immediately if the server is 
busy upon his arrival. The service time has an exponential distribution with 
parameter 4; when the service is provided to 7 customer(s) at a time, for 
i = 1,2. Moreover, we suppose that the service times are independent random 
variables and that there can be at most two customers waiting at any time. 
We define the states 


0: nobody is being served 
ni: there is (are) n customer(s) waiting and į customer(s) being served 


for n = 0,1,2 andi = 1,2. 
(a) Write the balance equations of the system. Do not solve them. 


(b) Calculate, in terms of the limiting probabilities, the probability that 
(i) the server is serving two customers at a time, given that he is busy, 
(ii) an arriving customer who enters the system will not be served alone. 


Question no. 10 

Customers arrive into a queueing system according to a Poisson process 
with rate A. There is a single server, who cannot serve more than one customer 
at a time. However, the larger the number of customers in the system is, the 
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faster the server works. More precisely, we suppose that the service time has 
an exponential distribution with parameter u(k) = roa u when there are k 
customers in the system, for k = 1,2,.... Moreover, we suppose that the 
service times are independent random variables and that A = 1 and p = 2. 


(a) Calculate the limiting probabilities mn, for n =0,1,... . 


(b) Let X(t) be the number of customers in the system at time t, for t > 0. 
Calculate E[.X(t) | X(t) < 2] when the system is in equilibrium. 


(c) Let To be the time spent in the system by a customer who arrived, at time 
to, while the system was empty. Calculate the expected value of To, given that 
the following customer arrived at time to + 1 and the customer in question 
had already left the system. 


Question no. 11 

Drivers arrive according to a Poisson process with rate to fill up their 
cars at a service station where there are two employees who serve at the 
exponential rates u and u2, respectively. However, only one employee works 
at a time serving gasoline. Moreover, there is space for only one waiting car. 
We suppose that 


e when the system is empty and a customer arrives, employee no. 1 fills up 
the car, 

e when employee no. 1 (respectively, no. 2) finishes filling up a car and 
another car is waiting, there is a probability equal to pı (resp., p2) that 
this employee will service the customer waiting to be served, independently 
from one time to another. 


Finally, we suppose that the service times are independent random vari- 
ables. 


(a) Let X(t) be the state of the system at time t. Define a state space in such 
a way that the stochastic process {X(t),¢ > 0} is a continuous-time Markov 
chain. 


(b) Write the balance equations of the process. 


(c) Calculate, in terms of the limiting probabilities, the probability that 

(i) an arbitrary customer entering the system will be served by employee 
no. 2, 

(ii) two customers arriving consecutively will be served by different em- 
ployees, given that the first of these customers arrived while there was exactly 
one car, being filled up by employee no. 1, in the system. 


Question no. 12 

We consider the queueing system M/M/1. However, the customers do not 
have to wait, because the server is able to serve all the customers at one time, 
at an exponential rate u, regardless of the number of customers being served. 
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Calculate 


(a) the limiting probability, mn, that there are n customers being served, for 
TOs A P 

(b) the variance of the number of customers being served when the system is 
in equilibrium. 


Question no. 13 

Suppose that customers arrive at a service facility with a single server 
according to a Poisson process with rate À. The server waits until there are 
four customers in the system before beginning to serve them, all at once. 
The service times are independent random variables, all having a uniform 
distribution on the interval (0,1). Moreover, the system capacity is equal to 
four customers. What fraction of time, m;, are there i customer(s) in the 
system, over a long period? 


Indication. Use the results on renewal processes. 


Question no. 14 

Suppose that the times between the arrivals of consecutive customers in 
a certain queueing system are independent random variables uniformly dis- 
tributed over the interval (0,1). The service time is exponentially distributed, 
with parameter u. Finally, the (only) server is able to serve all the customers 
at once, so that there is no waiting. Calculate the limiting probability that 
the server is busy. 


Indication. Use the results on renewal processes. 


Question no. 15 

We modify the M/M/1/4 queueing system as follows: the server always 
waits until there are at least two customers in the system before serving them, 
two at a time, at an exponential rate u. 


(a) Write the balance equations of the system. 
(b) Calculate the limiting probabilities, mp, for n = 0,1,2,3,4, in the case 
when A = u, where À is the average arrival rate of the customers. 
(c) With the help of the limiting probabilities calculated in (b), find 

(i) the probability that the system is not empty at the moment when the 
server has just finished serving two customers, 

(ii) the variance of the number X(t) of customers in the system (in equi- 
librium) at time t, given that X(t) < 2. 


Section 6.3 


Question no. 16 

Customers arrive according to a Poisson process with rate A at a bank 
where two clerks work. Clerk 1 (respectively, 2) serves at an exponential rate 
Hı (resp., u2). We suppose that the customers form a single queue and that, 
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when the system is empty, an arriving customer will go to clerk 1 (resp., 2) 
with probability pı (resp., 1 — pı). On the other hand, when a customer must 
wait, she will eventually be served by the first available clerk. We also suppose 
that an arbitrary customer can enter the bank only if there are no more than 
10 customers waiting in line. We say that the system is in state n = 0,2,... 
if there are n customers in the bank, and in state 1, (resp., 12) if there is 
exactly one customer in the bank and if this customer is being served by clerk 
1 (resp., 2). 

(a) Write the balance equations of the system. 


(b) In terms of the limiting probabilities, what is the probability that an 
entering customer will be served by clerk 1? 


Question no. 17 

In a certain garage, there are three mechanics per work shift of eight hours. 
The garage is open 24 hours a day. Customers arrive according to a Poisson 
process with rate A = 2.5 per hour. The time a mechanic takes to perform 
an arbitrary task is an exponential random variable with mean equal to 30 
minutes. 


(a) What proportion of time are all the mechanics busy? 


(b) How much time, on average, must a customer wait for his car to be ready? 


Question no. 18 

In a small train station, there are two counters where the travelers can buy 
their tickets, but the customers form a single waiting line. During the slack 
hours, only one counter is manned continuously by a clerk. When there is at 
least one customer waiting to be served, the second clerk opens his counter. 
When this second clerk finishes serving a customer and there is nobody wait- 
ing, he goes back to attending to other tasks. We suppose that the clerks both 
serve in a random time having an exponential distribution with parameter p 
and that, during the slack hours, the customers arrive according to a Poisson 
process with rate À. We also suppose that the slack period lasts long enough 
for the process to reach a stationary regime. The state X(t) of the system 
is defined as being the total number of customers present in the system at 
time t. 


(a) Calculate the limiting probabilities of the process {X(t),t > 0} if A < 2y. 


(b) Write the balance equation for the state 12 corresponding to the case when 
only the second clerk is busy (serving a customer). 


(c) What fraction of time is the second counter open? 


Question no. 19 

Customers arrive at a hairdresser’s salon according to a Poisson process 
with rate A = 8 per hour. There are two chairs, and the two hairdressers’ ser- 
vice times are exponential random variables with means equal to 15 minutes. 
Moreover, currently, there is no room where potential customers could wait 
for their turn to have their hair cut. 
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(a) The owner considers the possibility of enlarging the salon, so that she could 
install an additional chair and hire a third hairdresser. This. would increase 
her operation costs by $20 per hour. If each customer pays $10, would the 
enlargement be profitable? Justify. 


Indication. Calculate the average rate at which customers enter the salon. 


(b) Another possibility the owner considered consists of enlarging the salon to 
install a chair where one potential customer could wait to be served. In this 
case, the increase in operation costs would be equal to $5 per hour. Would 
this possibility be profitable? Justify. 


Question no. 20 
We consider the queueing system M/M/2/3 (see Example 6.3.2). 


(a) Write the balance equations of the system, and calculate the limiting 
probabilities m; if A = 2u. 

(b) Let T* be the total time that an entering customer will spend in the 
system. Calculate the expected value of T* if p = 1. 


(c) Suppose that the customers form two waiting lines, by standing at random 
in front of either server. Calculate, with u = 1, the average time that an 
arbitrary customer who enters the system, and finds two customers already 
present, will spend in this system if we assume that the number of customers 
in each queue is then a random variable having a binomial distribution with 
parameters n = 2 and p = 1/2. 


Question no. 21 

We consider a queueing system with two servers. The customers arrive 
according to a Poisson process with rate A = 1, and the system capacity is 
equal to four customers. The service times are independent random variables 
having an exponential distribution. Each server is able to serve two customers 
at a time. If a server attends to only one customer, he does so at rate u = 2, 
whereas the service rate is equal to 1 when two customers are served at the 
same time. 


Indication. If two customers are served together, then they will leave the 
system at the same time. Moreover, if there are two customers in the system, 
then one of the servers may be free. 


(a) Write the balance equations of the system. 


(b) Let T* be the total time that a given customer entering the system will 
spend in it. : 

(i) Calculate, in terms of the limiting probabilities, and supposing that no 
customers arrive during the service period of the customer in question, the 
distribution function of T*. 

(ii) Under the same assumption as in (i), does the random variable 7™ 
have the memoryless property? Justify. 
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Question no. 22 

Customers arrive according to a Poisson process with rate A = 5 per hour 
in a system with two servers. The probability that an arbitrary customer 
goes to server no. 1 (respectively, no. 2) is equal to 3/4 (resp., 1/4). The 
service times (in hours) are independent exponential random variables with 
parameters u = 6 and po = 4, respectively. A customer who goes to server 
no. 2 immediately leaves the system after having been served. On the other 
hand, after having been served by server no. 1, a customer (independently 
from one time to another) 


leaves the system with probability 1/2 
goes to server no. 2 with probability 2/5 
returns in front of server no. 1 with probability 1/10 


Moreover, there is no limit on the number of customers who can be in the 
system at any time. 

Let (n,m) be the state of the system when there are n customers in front 
of server no. I and m customers in front of server no. 2. 


(a) Calculate Tin,m), for all n,m > 0. 


(b) Calculate the average number of customers in the system at a large enough 
time instant, given that the system is not empty at the time in question. 


(c) What is the average time that an arbitrary customer who arrives in the 
system and goes to server no. 1 will spend being served by this server before 
leaving the system if we suppose that the customer in question never goes to 
server no. 2? 


Question no. 23 

Customers arrive according to a Poisson process with rate À outside a bank 
where there are two automated teller machines (ATM). The two ATMs are 
not identical. We estimate that 30% of the customers use only ATM no. 1, 
while 20% of the customers use only ATM no. 2. The other customers (507%) 
make use of either ATM indifferently. The service times at each ATM are 
independent exponential random variables with parameter p. Finally, there is 
space for a single waiting customer. We define the states 


0: the system is empty 
(n,m): there are n customers for ATM no. 1 and m customers 
for ATM no. 2, fori <n+m<2 
3: the system is full 


(a) Write the balance equations of the system. Do not solve them. 


(b) Calculate, in terms of the limiting probabilities, 

(i) the variance of the number of customers who are waiting to use an 
ATM, 

(ii) the average time that an arbitrary customer, who enters the system 
and wishes to use ATM no. 2, will spend in the system. 
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Question no. 24 

We consider a queueing system in which ordinary customers arrive accord- 
ing to a Poisson process with rate À and are served in a random time having 
an exponential distribution, with parameter u, by either of two servers. Fur- 
thermore, there is a special customer who, when she arrives in the system, is 
immediately served by server no. 1, at an exponential rate us. If an ordinary 
customer is being served by server no. 1 when the special customer arrives, 
then this customer is returned to the head of the queue. We suppose that 
the service times are independent random variables and that the special cus- 
tomer spends an exponential time (independent of the service times), with 
parameter Às, outside the system between two consecutive visits. 


(a) Suppose that if an arbitrary customer is returned to the queue, then he 
will resume being served as soon as either server becomes available. Define an 
appropriate state space, and write the balance equations of the system. 


(b) Suppose that the system capacity is c = 2, but that if a customer is 
displaced by the special customer, then she will wait, a few steps behind, until 
server no. 1 becomes available to resume being served by this server (whether 
server no. 2 is free or not). Define a state space such that the stochastic process 
{X(t),t > 0}, where X(t) represents the state of the system at time t, is a 
continuous-time Markov chain. 


(c) Suppose that the system capacity is c = 2 and that, if a customer is dis- 
placed by the special customer, then he will go to server no. 2 only if this 
server is free upon the arrival of the special customer in the system. Other- 
wise, he will wait, a few steps behind, before server no. 1 becomes available. 
Let K be the number of times that a given customer, who has started re- 
ceiving service from server no. 1, will be displaced by the special customer. 
Calculate P|K = 1] in terms of the limiting probabilities of the system (with 
an appropriate state space). 


Question no. 25 
Let N be the number of customers in an M/G/2/2 (loss) system after a 
time long enough for the system to be in stationary regime. 


(a) Calculate V[N | N > 0] if the service time, S, has a uniform distribution 
on the interval (0,1) and if the average arrival rate of customers in the system 
is A = 4. 

(b) Calculate V[N | X = 1/4] if S has an exponential distribution with pa- 
rameter 1/X, where X ~ U(0,1) and A = 2. 


Question no. 26 

Suppose that we modify the M/M/2 queueing system as follows: when a 
server is free, he assists (if needed) the other server, so that the service time, S, 
has an exponential distribution with parameter 2y. If a new customer arrives 
while a customer is being served by the two servers at the same time, then 
one the servers starts serving the new customer. 
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(a) Calculate the limiting probabilities if we suppose that A < 2u. 


(b) Suppose that the system capacity is c = 2 and that A = u. Calculate the 
average number of customers in the system in stationary regime, given that 
the system is not full. 


Question no. 27 

Let X(t) be the number of customers at time t > 0 in a queueing sys- 
tem with s servers and finite capacity c, for which the time 7 between two 
consecutive arrivals is a random variable such that f,(t) = 2te-* , for t > 0. 
We assume that the times between the arrivals of customers are independent 
and identically distributed random variables. Similarly, the service times are 
independent random variables having, for each server, the same probability 
density function as 7. We define the stochastic process {Y(t),t > 0} by 


Y(t) = X(g(t)) fort >0 


where g(t) is a one-to-one function of t. Find a function g such that {Y (t), 
t > 0} is an M/M/s queueing system with \ = u = 1 (and finite capacity c). 
Justify. 


Question no. 28 

We consider the loss system M/G/2/2. Suppose that the service times 
have independent exponential distributions with parameter ©, where O is a 
random variable such that 


fo(6) = = (26? +2941) for0<@<1 


Suppose also that the average arrival rate of customers is A = 1. Calculate, 
assuming they exist, the limiting probabilities 7;, for i = 0,1, 2. 


Question no. 29 

In the M/M/2 queueing system, we define the random variable S as being 
the first time that both servers are busy. Let 

F = exactly two customers arriving in the interval (0, ¢]. 
Calculate PIS < t| F]. 


Question no. 30 

A hairdresser and her assistant operate a salon. There are two types of 
customers: those of type I prefer to have their hair cut by the hairdresser but 
are willing to be served by her assistant, while those of type II want to be 
served by the assistant only. The type I (respectively, type II) customers arrive 
at the salon according to a Poisson process with rate à; (resp., A2). Moreover, 
the two Poisson processes are independent. Finally, the hairdresser (resp., 
the assistant) serves in a random time having an exponential distribution 
with parameter u, (resp., u2), and the service times are independent random 
variables. Answer the following questions, supposing that there is no room 
where potential customers can wait until being served: 
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(a) define a state space that enables you to answer part (c); 
(b) write the balance equations; 


(c) give, in terms of the limiting probabilities, 
(i) the average number of customers in the system; 


(ii) the average time that an arbitrary entering customer will spend in the 
system. 


Question no. 31 ; 
Redo the preceding question, supposing that the system capacity is infinite 
and that 


(a) only the potential type I customers are willing to wait until being served 
(those of type II go away if the assistant is busy), 


(b) only the potential type II customers are willing to wait until being served 
(those of type I go away if the hairdresser and her assistant are busy), 


(c) all the potential customers are willing to wait until being served. 


Question no. 32 

Suppose that in the queueing system M/M/2/c, with c = 3, server no. 1 
serves only one person at a time, at rate u1, while server no. 2 can serve one 
or two persons at a time, from any time instant, at rate H2. Moreover, when 
server no. 1 is free, an arriving customer’ will go to this server. 


(a) Let X(t) be the number of persons in the system at time t. Define a state 
space such that the stochastic process {X(t),t > 0} is a continuous-time 
Markov chain. 


Remark. Server no. 1 may be free while server no. 2 serves two customers at a 
time. That is, the two customers finish their service period with server no. 2. 
(b) Write the balance equations of the system. Do not solve them. 


(c) In terms of the limiting probabilities, what fraction of time does server 
no. 2 serve two customers at a time, given that she is busy? 


Question no. 33 

Redo the preceding question, supposing that the system capacity is instead 
c = 4 and that server no. 2 can serve one or two persons at a time (at rate 
u2) but only from the same time instant. 
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A.1: Distribution Function of the Binomial Distribution 
A.2: Distribution Function of the Poisson Distribution 
A.3: Distribution Function of the N(0, 1) Distribution 
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Table A.1. Distribution Function of the Binomial Distribution 


2 | 0 


p 


0 
1 
12 
0 
1 
2 
3 
0 
i 
2 
3 
4 
0 
1 
2 
3 
4 
9 
6 
T 
8 
9 
0 
1 
2 
3 


0.05 


0.10 
0.8100 
0.9900 


0.7290 
0.9720 
0.9990 


0.6561 
0.9477 
0.9963 
0.9999 


0.5905 
0.9185 
0.9914 
0.9995 
1.0000 


0.3487 
0.7361 
0.9298 
0.9872 
0.9984 
0.9999 
1.0000 


0.20 
0.6400 
0.9600 


0.5120 
0.8960 
0.9920 


0.4096 
0.8192 
0.9728 
0.9984 


0.3277 
0.7373 
0.9421 
0.9933 
0.9997 


0.1074 
0.3758 
0.6778 
0.8791 
0.9672 
0.9936 
0.9991 
0.9999 
1.0000 


0.0352 
0.1671 
0.3980 
0.6482 


0.25 
0.5625 
0.9375 


0.4219 
0.8438 
0.9844 


0.3164 
0.7383 


0.9493 


0.9961 


0.2373 
0.6328 
0.8965 
0.9844 
0.9990 


0.0563 
0.2440 
0.5256 
0.7759 
0.9219 
0.9803 
0.9965 
0.9996 
1.0000 


0.0134 
0.0802 
0.2361 
0.4613 


0.40 
0.3600 
0.8400 


0.2160 
0.6480 
0.9360 


0.1296 
0.4752 
0.8208 
0.9744 


0.0778 
0.3370 
0.6826 
0.9130 
0.9898 


0.0060- 


0.0464 
0.1673 
0.3823 
0.6331 
0.8338 
0.9452 
0.9877 
0.9983 
0.9999 


0.0005 
0.0052 
0.0271 
0.0905 


0.50 
0.2500 
0.7500 


0.1250 
0.5000 
0.8750 


0.0625 
0.3125 
0.6875 
0.9375 


0.0313 
0.1875 
0.5000 
0.8125 
0.9688 


0.0010 
0.0107 
0.0547 
0.1719 
0.3770 
0.6230 
0.8281 
0.9453 
0.9893 
0.9990 


0.0000 
0.0005 
0.0037 
0.0176 
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0.9873 
0.9977 
0.9997 
1.0000 
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0.2173 
0.4032 
0.6098 
0.7869 
0.9050 
0.9662 
0.9907 
0.9981 
0.9997 
1.0000 


0.0000 
0.0005 
0.0036 
0.0160 
0.0510 
0.1256 
0.2500 
0.4159 
0.5956 
0.7553 
0.8725 
0.9435 
0.9790 
0.9935 
0.9984 
0.9997 
1.0000 
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Table A.2. Distribution Function of the Poisson Distribution 


Le 


0 
1 
2 
3 
4 
9 
6 
T 
8 
9 
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Table A.3. Distribution Function of the N(0,1) Distribution 


0.5000 0.5040 0.5080 


0.5398 
0.5793 


0.5438 
0.5832 


0.5478 
0.5871 


0.6179 0.6217 0.6255 


0.6554 


0.6915 
0.7257 
0.7580 
0.7881 
0.8159 


0.8413 
0.8643 
0.8849 
0.9032 
0.9192 


0.9332 
0.9452 
0.9554 
0.9641 
0.9713 


0.9772 
0.9821 
0.9861 
0.9893 
0.9918 


0.9938 
0.9953 
0.9965 
0.9974 
0.9981 


0.6591 


0.6950 
0.7291 
0.7611 
0.7910 
0.8186 


0.8438 
0.8665 
0.8869 
0.9049 
0.9207 


0.9345 
0.9463 
0.9564 
0.9649 
0.9719 


0.9778 
0.9826 
0.9864 
0.9896 
0.9920 


0.9940 
0.9955 
0.9966 
0.9975 
0.9982 


0.6628 


0.6985 
0.7324 
0.7642 
0.7939 
0.8212 


0.8461 
0.8686 
0.8888 
0.9066 
0.9222 


0.9357 
0.9474 
0.9573 
0.9656 
0.9726 


0.9783 
0.9830 
0.9868 
0.9898 
0.9922 


0.9941 
0.9956 
0.9967 
0.9976 
0.9982 


0.5120 
0.5517 
0.5910 
0.6293 
0.6664 


0.7019 
0.7357 
0.7673 
0.7967 
0.8238 


0.8485 
0.8708 
0.8907 
0.9082 
0.9236 


0.9370 
0.9484 
0.9582 
0.9664 
0.9732 


0.9788 
0.9834 
0.9871 
0.9901 
0.9925 


0.9943 
0.9957 
0.9968 
0.9977 
0.9983 


0.5160 
0.5557 
0.5948 
0.6331 
0.6700 


0.7054 
0.7389 
0.7704 
0.7995 
0.8264 


0.8508 
0.8729 
0.8925 
0.9099 
0.9251 


0.9382 
0.9495 
0.9591 
0.9671 
0.9738 


0.9793 
0.9838 
0.9875 
0.9904 
0.9927 


0.9945 
0.9959 
0.9969 
0.9977 
0.9984 


0.5199 
0.5596 
0.5987 
0.6368 
0.6736 


0.7088 
0.7422 
0.7734 
0.8023 
0.8289 


0.8531 
0.8749 
0.8944 
0.9115 
0.9265 


0.9394 
0.9505 
0.9599 
0.9678 
0.9744 


0.9798 
0.9842 
0.9878 
0.9906 
0.9929 


0.9946 
0.9960 
0.9970 
0.9978 
0.9984 


0.5239 
0.5636 
0.6026 
0.6406 
0.6772 


0.7123 
0.7454 
0.7764 
0.8051 
0.8315 


0.8554 
0.8770 
0.8962 
0.9131 
0.9279 


0.9406 
0.9515 
0.9608 
0.9686 
0.9750 


0.9803 
0.9846 
0.9881 
0.9909 
0.9931 


0.9948 
0.9961 
0.9971 
0.9979 
0.9985 


0.5279 
0.5675 
0.6064 
0.6443 
0.6808 


0.7157 
0.7486 
0.7794 
0.8078 
0.8340 


0.8577 
0.8790 
0.8980 
0.9147 
0.9292 


0.9418 
0.9525 
0.9616 
0.9693 
0.9756 


0.9808 
0.9850 
0.9884 
0.9911 
0.9932 


0.9949 
0.9962 
0.9972 
0.9979 
0.9985 


0.5319 
0.5714 
0.6103 
0.6480 
0.6844 


0.7190 
0.7517 
0.7823 
0.8106 
0.8365 


0.8599 
0.8810 
0.8997 
0.9162 
0.9306 


0.9429 
0.9535 
0.9625 
0.9699 
0.9761 


0.9812 
0.9854 
0.9887 
0.9913 
0.9934 


0.9951 
0.9963 
0.9973 
0.9980 
0.9986 
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| z | +0.00 +0.01 +0.02 +0.03 +0.04 +0.05 +0.06 +0.07 +0.08 +0.09 


0.5359 
0.5753 
0.6141 
0.6517 
0.6879 


0.7224 
0.7549 
0.7852 
0.8133 
0.8389 


0.8621 
0.8830 
0.9015 
0.9177 
0.9319 


0.9441 
0.9545 
0.9633 
0.9706 
0.9767 


0.9817 
0.9857 
0.9890 
0.9916 
0.9936 


0.9952 
0.9964 
0.9974 
0.9981 
0.9986 
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0.9987 
0.9990 
0.9993 
0.9995 
0.9997 


0.9998 
0.9998 
0.9999 
0.9999 
1.0000 


0.9987 
0.9991 
0.9993 
0.9995 
0.9997 


0.9998 
0.9998 
0.9999 
0.9999 
1.0000 


0.9987 
0.9991 
0.9994 
0.9995 
0.9997 


0.9998 
0.9999 
0.9999 
0.9999 
1.0000 


0.9989 
0.9992 
0.9994 
0.9996 
0.9997 


0.9998 
0.9999 
0.9999 
0.9999 
1.0000 
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0.9989 
0.9992 
0.9994 
0.9996 
0.9997 


0.9998 
0.9999 
0.9999 
0.9999 
1.0000 


0.9989 
0.9992 
0.9995 
0.9996 
0.9997 


0.9998 
0.9999 
0.9999 
0.9999 
1.0000 


0.9990 
0.9993 
0.9995 
0.9996 
0.9997 


0.9998 
0.9999 
0.9999 
0.9999 
1.0000 





Appendix B: Answers to Even-Numbered 
Exercises 


Chapter 1 


2. (a) 0.0062; (b) ~0.80. 
4. 1 — 2/8 (~0.6073). 

6. 379/2187 (~0.1733). 

8. 244/495 (~0.4929). 

10. 1/3. 


12. 
2 C 


c e&€—(1+4+c) 
16. (a) Y ~ U[0, 1]; (b) g(x) = 2Fx (x) +1. 
18. (a) 2/3; (b) 1/3; (c) 8/3. 
20. (a) =0.3591; (b) =0.3769; (c) ~0.1750. 
22. (a) =0.4354; (b ) pb 80; (c) 81.5%. 
24. (b) (i) 0.1837; ( T2. 
26. (a) 


1 
Fyi,¥2(Y1, Y2) = ‘af 552 (5y7 + 2y2 — 6y1 Ye — 2y + 2p D 
for all (y1, y2) € R?; (b) 30? 
32. (b) V[X]/2. 
34. fzix(z | £) = fy (z — zx). 
36. (a) 
(ker Mk) T bs 5 forzeR 
z? + (dopa &) 
(b) fz(z) does not tend to a Gaussian density. The central limit theorem does 
not apply, because V[X;,] = 00 V k (and, actually, E|X;| does not exist). 
38. (a) Y/2; (b) 1/2. 
40. (a) 1/8; (b) 7/144; (c) ~ 0.9667. 


fz(z) = 
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42. (a) 
1 Pe a! 1 l 
= — -e ~“ |- +— if x >0 


£ 

(b) e~*; (c) 1/¥*; (d) œ. 

44. (a) 
0 ify <z 

Fyjx (y | z1) = fre sy<l 

lify>1 

(b) 3(1 +27); (c) 1/45; (d) 1/30. 

46. (a) 3/4; (b) 13/24; (c) 0; (d) X? + 3X1 4 5. 

48. 1 —e7! (~0.6321). 

50. 





_ 46)\2 
epf -RIE for x > 0 


54. (a) Xı + X2 + 4; (b) (X1 + X2 + X3); (c) 1/18. 


1 
fixre Xo) S ona 


Chapter 2 


2. (a) p(1 — p); (b) 0 ifn Am and 1 ifn =m. 

4. t/(2x*). 

6. The increments are not independent, but they are stationary. 
8. (a) 


lng 


Fait) = fy -F 


(b) E[X(t)] = 1/(1 + t) and Rx (ti, te) = E[X(t1 + t2)]. 

10. No, since E[X (t) (= (1 — e7*)/t) depends on t. 

12. No, because E|Y(t)] (= t) depends on t. 

14. E[X(t)X(t+s)| = E[Y°t(t+ s)|] = t(t + s)/7 # Rx(s). Consequently, the 
process is not WSS and therefore not SSS. 

16. Sy (w) = 2(1 — cosw)Sx (w). 

18. h(O)q(t). 

20. Cx (ky, k2) = Cx (ke — kı) = ke — ki a 0 and Cx (ke — kı) = pil — p) if 
kə — kı = 0. Since my (k) = p, the process is WSS. We have that Cx (0) < oo 
and liMjkj—=oo Cx (k) = 0. It follows that the process is mean ergodic. 

22. Yes, because Cx (0) = Rx(0) = 1 < œ and limjsj—=o Cx (s) = 0. 

24. c/2T. 

26. (a) e"; (b) (Y, Z) ~ N(uy = 0, uz = 0;0} = 2,03 = 2; p = 0). 

28. (a) and (b) No, because E[Y (t)] (= e~*) is not a constant. 

30. m(yo; to) = yo/2 and v(yo;to) = y3. 

32. (a) N(0,2); (b) 2e78*; (c) 4(1 + e789). 

34. (a) N(O,1); (b) U(1, 2). 


) = for x € (0, 1] 
tr 
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Chapter 3 


2. (a) 1/4; (b) (i) 0; (ii) no, because ETY (t)] (= t/2) is not a constant; (iii) 1/2. 
4. 11/12. 


6. 2/3. 

8. 0.1465. 

10. 
1/4 1/2 1/4 
1/21/20 
1/4 1/2 1/4 

12. 2/3. 


14. (a) Aperiodic, because it is irreducible and poo > 0; (b) 
i 
T; = (2) (ER) for i =0,1,2,3,4 
q/ \1—(p/q) 


22. (b) q — p if p < 1/2; 0 if p = 1/2; p — q if p > 1/2; (c) 0 if p < 1/2; 
(q/p)*—*[1 — (q/p)|, for k = 1,2,..., if p > 1/2. 
24. (a) A single class, which is recurrent and aperiodic; (b) m; = 1/5, for all 4; 


(c) no. 
28. (b) 
Í 1 
we (=) 
Aa 2 1 


aka “8! aaah 2 aaee 


32. (a) 1 (because the chain is irreducible and poo > 0); (b) mo = 3/7, 71 = 
2/7, = m3 = 1/7; (c) (i) 0.1716; (ii) 1. | 


34. (a) P(™ = Pl), where 
lepoop 
P®%=| 0 10 


t=p0p 


30. (a) 0 <a <1; (b) 


Ko 


if n is even, and P™ = P if n is odd; (b) d = 2; (c) (i) mo = (1 — p)/2; (ii) no, 
because this limit does not exist (since d = 2). 
36. (a) mo = 2/3 and 7; = (1/4)’, for j =1,2,...; (b) 1. 
38. (a) To = 3 = 1/0, n; = na: = 1/3; (b). d = 2; (ce) 2/3. 
40. (a) (i) 
i 0 0 0 0 
1/41/21/4 0 0 
0 1/41/21/4 0 
0 O 1/41/21/4 
0 0 0 0 1 


366 Appendix B: Answers to Even-Numbered Exercises 


(ii) yes; rå = mj = 1/2 and m+ = 0, for j = 1,2,3; (b) 1/4. 

42. (a)0<a<1,0< 6 <1; (b) to = T2 = 2/5 and mı = 1/5; (c) (i) 2/3; 
(ii) 1/4 + a/2. | 

44. (a) a#1, B #0; (b) ro = 1/9, mı = 5/9, and m2 = 1/3; (c) a = 1/2 and 


B = 1; (d) 3. 
46. 
. (rn) _ jiifg=l1 
bare eC re =Oor2 
and 
2/3 if j =0 
lim D = lim py) =< 0 ifj=1 
ss Pe 1/3 if j = 2 


48. (a) 1/2; (b) we have 
2(mt)* + (xg)? — 4r +1 =0 


We can check that the value (1/2)? = 1/4 is not a solution of the equation, 
so that n 4 72. | 


50. 
am) flifj=1 
im, p y o otherwise 
and 

2/5ifj=0 
i (n) p (n) y (m j 0 ifj=1 
n pop S p = in a SFS 
2/5 if j = 3 


52. (a) -1 + V2 (~0.4142); (b) for example, pp = po = p3 = 1/3 and pı = 0. 
54. (a) The state space is {0,1,2}, and the matrix P is given by 


(1 — pi)(1 — po) pı + p2 — 2pipe pipe 
i-p p 0 
(1 — p1)(1 — po) pı + po — 2pıp2 pıp2 


where p; := e7, for i = 1,2, and 


ee a PEE 
ala Ls et ae, is aoe 


(b) Yes, by the memoryless property of the exponential distribution. We cal- 
culate 
(1 — p1)(1 — p2) pı(1 — p2) (1 — pi) p2 pipe 
P= lpi Pi 0 0 
l — p2 0 P2 0 
(1 — pi)(1 — p2) pı (1 — p2) (1 — pi) pe pipe 


56. (a) There are 3? = 9 possible states: 0 = (0,0), 1 = (0,1), 2 = (0,2), 
3 = (1,0), 4 = (1,1), 5 = (1,2), 6 = (2,0), 7 = (2,1), 8 = (2,2). We find that 
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0 0 
00 
0 0 
0 0 
p O 
0 p 
p 0 
P p 
p 


where p = 1/4; (b) 7; = 1/9, for all i. 
58. n. 
00a PN S= s ond PIN Esa (12) fort 253.50 9 
(b) (n = 1)/(24); (c) G(a=n- 1,A = 24). 
62. (a) 
1/N 1/N1/N...1/N 1/N 1/N 
0 2/N1/N...1/N 1/N 1/N 
PS wae. kOe wee do aes ae ota 
0 0 0... 0 (N-1)/N1/N 
0 0 O ... O 0 1 


(b) (i) We have 


1 1 
mi = 1+ amit ap Mi +... + my) for i = 1,...,N —1 


We find, with my = Q, that m; = N; (ii) m; = E[Geom(p := 1/N)| = N, for 
Ce TE = 1. 

64. (a) ~0.00002656; (b) 20.8794; (c) ~—$20. 

66. (a) 


—4| A790: 0 0 1/2 


-2 1/2 0 0 1/2 
P= -1 1/2 0 1/2 
0 1/2 0 1/2 


1 1/2 0 1/2 


(b) m4 = T- = 1/16, r_; = 1/8, and To = T1 = 72 = 1/4; (c) 3/8. 

68. ni = Pj V ve 

70. (a) 9/4. 

WAC) Rio. 

76. (a) 30/37; (b) 16/37; (c) 4/19. 

78. (a) 0.1170; (b) ~0.0756. 

82. (a) 1+ At; (b) zo = 1 and mẹ = 0, for k = 1,2,...; (©) (At)? /(1 + At)?. 
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84. (b) e~*(e?4 — 1); (c) the 7;’s do not exist, because {X(u),u > 0} is a 
pure birth process. 

86. (a) a = 0; (b) mo = 10/27, mı = 8/27, and m2 = 9/27. 

88. (a) No, because the time 7 that the process spends in either state does not 
have an exponential distribution, since P{[7 > t] = e~ + 2e% (b) =0.2677. 
90. 12(1 — e7%)/(3 — 2e7?). 

92. (a) Ao = 2A and Ay = Un = A, for n = 1,2,...; (b) the 7;’s do not exist. 

94. (a) Ao = 2A, Ay = A/2, pı = 3A/2, and u2 = 2A; (b) 1/2; (c) mo = 3/8, 
mı = 1/2, and m2 = 1/8. 

96. 


c—k ; a 
en 5 (Aer rt 4 u) (1 =) e Henne)” k 
H) 


98. (a) We define the states 


0: the machine is functioning 

1: failure of type 1 

2: failure of type 2; first repairing operation 
3: failure of type 2; second repairing operation 


(b) u/[u + (2 — a)àl]. 
Chapter 4 


2. (a) 0; (b) t+min{t?,t+r}+min{t, (t+7)?}+t?; (c) (i) yes; (ii) no, because 
Cov[X (t), X(t + 7)| depends on t; (iii) no, since Cov| X(t), X(t + 7)] 4 o7t; 
d) 


ft? if0<t<1 
PB(t),B(t?) = $ 4-1/2 ft>] 


4. (a) No, because X(t) > 0; (b) no, since E[X*(t)] (= t) is not a constant. 
6. 4. 

8. Brownian motion with g? = 2. 

10. (a) No, because Y (t) > 0; (b) 1; (c) 2s/t; (d) no, because Cov[Y (s), Y (t)] 
is not a function of | t — s |. 

12. (a) B(j + 1 — i) — S(j — i), where @ is the distribution function of the 
N(0, 1) distribution; (b) ~0.4659. 

14. (a) 


3 (t+ e- mint +e,t+s)}) 
(b) (i) yes; (ii) yes; (iii) no, because Cy (t,t + s) Æ ot; (iv) yes. 


16. (a) N(0,2(1 + e~*)); (b) (i) E[U(t)] = 0 and V[U(t)] = 2(t + e~* — 1); 
(ii) yes. 
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18. (b) No, Pe we have that a )] = exp{( + 40°)t}, which depends 
on t (if u # —}40°); if w = —40°, then E[Y(t)] = 1, but E[Y (t + s)Y(t)| 
(=e?) is not a function of s. 

20. (a) 0; (b) yes, because X and Y are Gaussian random variables and their 


covariance is equal to zero. 
22. E|Y (t)| = 0 and 


2 
Cy(s,t) = > (exter) — gi) for s,t > 0 


24. (a) N(0, 11/3); (b) 


fr, (t) = ex p{-aat for t > 0 


3 


26. (a) E[Y (t)| = 0 and 


3 42 5)2 
Cov[Y (t), Y (t + s)| = ; SH (s | | 





(b) (i) Yes; (ii) no, because Cov[Y(t), Y(t + s)|] depends on t; (iii) no, since 
Cov[Y (t), Y(t + s)] # t[1 — (t + s)]; (c) = 2P[N(0,1) > V12d]. 
28. We find that 


od d2e-t d —t/2 deh 
—_ Se a rere amam ey > 
mot =- {an} a (recs) 
30. (a) 0; (b) 1 + 2t; (c) no, because Cov|Z(t), Z(t + s)] depends on t; (d) 


= _2(d—z) (d—z)? 
fryz)(t) = ie aaah TE for t > 0 


Chapter 5 


2. (a) e~?° (0.0821); (b) ~0.0028; (c) 1/3; (d) 
10, 
Cx (t1,t2) = —— min{t,, t2} 
tita 


4. (a) (i) e~? (0.1353); (ii) =0.1839; (iii) ~0.3233; (b) no, because E[Z(t)| = 
exp{t(e~+ — 1)}, which depends on t. 

6. e7°. 

8. As. 

10. (a) (1 — e71)?; (b) 1.64. 

12. (a) 0.3535; (b) ~0.0119. 
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14. (a) 3/64; (b) #2. 

16. (a) (i) Mean = 15, variance = 240; (ii) mean = 10, variance = 520/12; (b) 
~14.785. 

18. 1/(2V)). 

20. PIN*(8) = 1] = o(6). 

22. (a) Let y := A(e° — 1). We find that 


-a en 
ole — 1) ifs £0 
2y 
1 ifs=0 


My t+s)(s) = 


(b) E[N(t + S)] = A(t +1) and V[N(t + S)] = A(t +1) + 44°. 

24. (a) e7+(b — a)/15; (b) 1/16; (c) 1/16. 

26. (a) 0.6723; (b) 1/2; (c) 1/3. 

28. (a) U(0, s); (b) 5e7* (0.0916); (c) 10/3. 

30. (a) (i) 1/(1 — t), for 0 < t < 1; (ii) 1/2; (b) (ii) P[N(0, 1) < 2] (~0.9772). 
32. (a) 0.5590; (b) 4.75; (c) ~0.0409; (d) ~0.0328. 

34. ~0.9817. 

36. 1 — e7? (142A + 3)?). 

38. +0.217, 

40. (a) n!/n”; (b) 2(1 — t) if 0 <t < 4, and 2 (t — 5) ifi<t<l. 
42. {X(t),t > 0} is a Poisson process with rate À. 

44. Yes; we have that p;i; = 0 if j < i and 


F \vi- vi 
Pig = @ “7 


(V7 — vi)! 


if j > i, where i,j € {0,1,2,4,9,...}. 

46. (a) 1/3; (b) (i) 2x e7% , for x > 0; (ii) ~0.45; (c) 9/5. 
48. (a) ~0.2565; (b) 1/5. 

50. (a) 0.56; (b) 3/16. 

52. (a) 1 — 2e7! (~0.2642); (b) 0.75 t + 0.5 t?. 

54. (a) 1/3; (b) 588; (c) 232. 

56. (a) ~0.4562; (b) 2; (c) 2; (d) 1/2, if s € (0, 1] or (2, 3]. 
58. (a) la(s? + 1)/1n2; (b) ~0.6068. 

60. (a) 12t/(1 — e~*); (b) PIN(0, 1) < 0.69] (~0.7549). 
62. (a) Mean = 8 and variance = 40; (b) 401; (c) 


0 ifs <6 


| 2 feces 
eo See ges 
eog 
1 ` ifs>8 





64. (a) (i) Mean = variance = 21; (ii) mean = variance = 24; (b) ~0.1562. 
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66. (a) Ay/(2A1 + Az); (b) Ait; (c) 1 — e7? (0.9323). 
68. (a) Gla = 2, A); that is, 


Tat) = Mre~*7 forr > 0 and i= 1,2,... 
(b) we find that 


a AD . (at)?! 
P e E E a for n =0,1,... 
RS (2n)! 5 (2n +1)! =e 


(c) 3/(2A). 

70. (a) u/[u + (2 — aA]; (b) (3/4)(2 — a) /(3 — a). 
72. (a) G(a = 2,2); (b) ~0.81; (c) ~1.17. 

74. (b) ~1/2; (c) 1/2. 

76. (a) We find that 


my(t) = epf =} (eteri — 5) —1 for1/2<t<1 
(b) P{N(O, 1) < 0.48] (~0.68); (c) 5/8. 
78. (a) We calculate 


1 t 
my(t) =+ (epf }-1) for0<t<1 


(b) e — e!/? (~1.07). 

80. No; since the number of events per time unit increases linearly, the time 
between two consecutive events is (on the average) shorter and shorter, so 
that the random variables 7 are not identically distributed (they are not 
independent either). 

82. fy > 0. 

84. (a) 1/(2a” + 1); (b) 0. 





Chapter 6 


2. (a) N = 11/15 and Ng = 4/15; N # Ng + 1 because ro # 0; (b) 1/15. 
4, (a) Ñ = A/p and Ng = e™™ + + à — 1; (b) 1+ G(e 4 — 1). 
6. (a) We have 


state j departure rate from j = arrival rate to 7 


0 2A To ZUTI 

1 (2A + u) Ti = U T2 + 2A To 

2 (A+ u) mo = u T3 +2À n 

> 3) (A + H) Tk = H Akay + À Tk- 


372 Appendix B: Answers to Even-Numbered Exercises 


(b) 1/2; (c) 3/8. 
8. (a) We have 


state 7 departure rate from j = arrival rate to 7 


0’ 10 7 = 5 To 
0 (10 +5) mo = 10 76 +5 (m1 + m2 +73) 
1 (10 +5) 7 = 10 zo 
2 (10 +5) m2 = 10 m 
3 5 73 = 10 m2 


(b) To = ig To = 21T, y= 4/21, T2 = ape and 3 = 16/63; (c) 5/21. 
10. (a) tm = (n + 1)(1/2)"*?, for n =0,1,...; (b) 10/11; (c) 20.4180. 
12. (a) tm = (x45) (x) for n= 0,1,...3 (b) “(A+ y). 

14. 4/(u + 4). 

16. (a) We have 


state 7 departure rate from j = arrival rate to 7 


0 À To = H1 Ti, + H2 Tig 

lı (A+ H1) Ti, = pid To + H2 Te 

l> (À + H2) Fiz = (1 — pi) To + fy N2 

2 (A + fi + H2) T2 = A (T1, + Tia) + (Hi + H2) T3 
3<n<12 (A + Hi + U2) Tn = À Tn-1 + (H1 + H2) An41 

13 | (441 + H2) T13 = A T12 


(b) the probability requested is given by 


ro p en Greer =a) Som 


18. (a) We find that 


2—A 
24+A 





= Te forn =1,2 
no = and Tmn = i gn—1 0 Or n = i, d,s 


(b) (A+ u) Ti, = u 79; (c) the fraction of time that the second counter is 
open is given by 
r?(2u — A) x 
2u(A + u)(2u +A) p(2u +À) 
20. (a) We have 


state 7 departure rate from 7 = arrival rate to 7 


À To = UT 
(A+ p) Ti = À To + 2p T2 
(A + 2u) T2 = À T1 + 2p T3 
2u T3 = À Tə 


Whe © 
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We find that if À = 2y, then mp = 1/7 and m1 = 72 = 73 = 2/7; (b) 1.2; (c) 2. 
22. (a) We find that 


Iny] 35\™ /13 
B E EN ad for all n,m > 0 
nue) a (3) 6 (i) pee 


(b) ~5.41; (e) 1/5. 
24. (a) We define the states 


0: the system is empty 

n: there are n > 1 ordinary customers in the system 

Ns: there are n > 1 customers in the system and the 
special customer is being served 


The balance equations of the system are the following: 


state j departure rate from j = arrival rate to j 


0 (A + As) To = U T1 + Us T1, 
1 (A +A, +H) Ti = À To + 2p Tat Us Te, 
ls (À + Hs) Ti, = Às To +H T2, 
n22 (A A ga 2u) Nn = À nn—1 + 2H Tn+1 + Hs n+l). 


Ns (n > 2) (A+ lts + H) Tr, = Às Mn-1 +A Tn-1), t H Tin+1)s 


(b) we define the states (m,n) and (ms, n), where m € {0,1,2} (respectively, 
n € {0,1}) is the number of customers in front of server no. 1 (resp., no. 2) 
and s designates the special customer; there are therefore eight possible states: 
(0,0), (1,0), (0, 1), (140), (1,1), (1s 1), (26,0), and (25, 1); (c) we can use the 
state space of part (b); we find that the probability requested is given by 


Às \ (1,0) +( As (A) ae 
H+ Às) Tao ttan  \et+As) \u+As/ Ta,0) + 7,1) 


n 
26. (a) m = 1 — A 5 and Th = i) To, for n = 1,2,... (this model is 


equivalent to the M r/M/ 1 model with service rate equal to 2u rather than u); 
(b) 1/3. 

28. To = 1/4 and m7 = T2 = 3/8. 

30. (a) We define the states (i, j), for i,j = 0,1, where i (respectively, j) is 
equal to 1 if a customer is being served by the hairdresser (resp., the assistant), 
and to 0 otherwise; (b) the balance equations of the system are the following: 





state j departure rate from j = arrival rate to 7 


(0, 0) (A1 + A2) 70,0) = Ha 71,0) + H2 70,1) 
(1,0) (Ar + Ag + p1) 71,0) = Ar 70,0) + H2 74,1) 
(0, 1) (Ai + u2) 70,1) = A2 70,0) + Mi 7,1) 


(1,1) (Hi + L2) Ta) = At Teo) + (At + A2) 70,0) 
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(c) (i) N = m3) + 71,0) +2 7,1); (ii) T = N/Ac, where 
Ne = At (1 — 74,19) + A2 (70,0) + 7(1,0)) 

32. (a) We define the states 


0: the system is empty 
1,: there is one customer in the system, being served by server no. 1 
lg: there is one customer in the system, being served by server no. 2 
2: there are two customers in the system, one per server 
22: there are two customers in the system, being served by server no. 2 
3: there are three customers in the system, two of them being served 
by server no. 2 


(b) the balance equations of the system are given by 


state 7 departure rate from j = arrival rate to J 


0 À TO = Hi Ti, + H2 (Tig + T23) 
li (A+ Hi) Ti, = À To + Me (T2 + T3) 

lo (A + u2) Ti, = H1 Te 

2 (A + ui + u2) m2 = A (m1, + ni3) 

25 (A+ u2) T2, = H1 T3 

3 (ui + u2) 73 = A (T2 + Te.) 


(c) the fraction of time requested is (m2, + 73)/(1 — To — T1, )- 
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